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ABSTRACT 



Five background papers were produced for the Research, 



Development and Dissemination System Committee of the National Educational 
Research Policy and Priorities Board in preparation for the proposed 
reauthorization of the Department of Education's Office of Educational 
Research and Improvement. The first paper, "Shaping the Future of Educational 
Research, Development, and Communication," describes and assesses the current 
system of educational research and development. The second paper, "The 
Educational Research, Development, and Dissemination System: An Analytic 
Mapping, " explores the characteristics of the educational research and 
development system and the issues involving it. The third paper, "The Needs, 
Demand for, and Supply of Educational Research, Development, and 
Dissemination, " provides a more detailed analysis of the nature of the 
research, development, and dissemination (RD&D) work required and demanded 
versus what is actually delivered. The fourth paper, "Comparison of Selected 
Federal RD&D Styles," considers alternative styles for the conduct of federal 
RD&D, focusing on the cognitive science program of the Office of Naval 
Research and the learning disabilities and reading program of the National 
Institute of Child Health and Human Development. The final paper, "A Review 
of the Federal Role and the Department of Education Structure, " summarizes 
the federal role in research, specifically in educational research, and the 
structure for fulfilling the research role in the U.S. Department of 
Education. (Contains 11 tables, 1 chart, 3 figures, and 47 references.) (SLD) 
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INTRODUCTION 



The long-standing national goal for a well-educated citizenry faces new, more 
challenging demands for the 21 st century just ahead. Those demands are led by the increase in 
the knowledge and skills required for a larger share of our working population by job markets. 
Those demands are made clear by continuing stories in our newspapers and other media of 
difficulties employers face in finding enough well-qualified and motivated workers to fill 
available jobs, which results in sending work abroad or diminishing national economic 
performance. These media reminders are underscored by disturbing academic analyses which 
point to the falling real income of high school graduates and the inability of almost half of those 
graduates to score sufficiently well on standard exams in reading and math to gain employment 
in a modem automobile plant.* 

The differences in earnings between high school and college graduates continue to grow, 
but only in part because job content requires college level courses. The needed reading, math, 
computer and problem solving knowledge and skills can be and are taught and learned in 
elementary and secondary curricula; or at most, can be gained with one to two years of additional 
instruction for many jobs with good economic futures. Employers are also interested in potential 
employees with the so-called “soft” skills of teamwork and communication, all skills that can be 
taught in the elementary and secondary system. Employer demand for college education is, 
unfortunately, explained in part by the perceived limitations of the elementary and secondary 
system in producing an adequate knowledge and skill base for the workplace; and it is further 
fueled by parental ambitions for their children. 

The growing demand for a stronger level of knowledge and skill goes beyond the 
demands of employment. To function in any role in the 21 st century society, individuals will 
need to be able to read, figure and solve problems on their own at higher levels than before to 
conduct the ordinary, but increasingly complex functions of life and participate in the activities 
in their neighborhoods and the world at large. These growing demands call on all institutions 
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and individuals, and perhaps our educational system most of all, to raise the level of educational 
performance to meet the challenge. 

The challenge posed by “raising the bar” of acceptable educational performance 
standards and outcomes is paralleled by two important contextual trends, which add to the 
difficulty. The first of these is the rapid increase in the number of students “at risk” in school 
districts that are least able to marshal the human and financial resources to meet these growing 
demands. The demographic and geographic characteristics of the projected growth in the youth 
population over the next 30 years suggest that virtually all of it will be concentrated in these “at 
risk” areas. But one should not suppose that the challenge is confined to “at risk” kids in 
deteriorated urban school systems. As the recent Third International Mathematics and Science 
Study (TIMSS) makes clear, deficiencies in math skills and knowledge extend to all students at 
all levels of relative proficiency, including well-financed districts with mostly white students. 

A second important contextual trend adding to the challenge of an effective educational 
response is the accelerating onrush of information technology. Already reaching into the lives of 
students, the new technologies are increasingly shaping formal education, for better or worse, 
and reemphasizing disparities between the “haves” and “have-nots.” This challenge is not just 
about computer hardware and limited resources, vital as these factors are; it is also about the 
critical need to plan and integrate new technology into teaching and curricula, so as to expand 
and extend student learning. 

These interacting challenges and issues represent a problem of immense national 
significance. The national educational enterprise in its many forms is widely and correctly 
understood to be a central provider of the knowledge, skills and perspective necessary to the 
success of our economy and the well being of our society. It is also part of the ethos that binds us 
together as a community and to bridge our many differences. The meeting of the new challenges 
requires more than good will, energy and resources. It also requires putting our knowledge to 
work and expanding our knowledge to meet the challenges posed by important trends. Trying to 
implement our hopes, our goals, or worse, our slogans without careful research, testing and 
development is likely to increase our frustrations without improving our performance. 
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Careful research, development and effective communication of findings have led to 
important contributions to sound educational practice. Examples of such contributions include: 

• the growth and utilization of cognitive science, particularly as applied to reading; 

• the improved understanding of the potential and limitations of measurement, 
assessment and testing; 

• a better understanding of the organization of learning experiences; 

• the necessary alignment of objectives, curriculum, teacher training and assessment 
for effective education policy and performance; 

• and the importance of self-assessment to the achievement of comprehensive school 
reform. 

The less cheerful news is how much we need to learn, particularly about the core processes of 
instruction and learning. These are the problems which parents, students and teachers want 
solved. The circumstances we face in the decades ahead require wise and substantially larger 
investments in our knowledge base to keep pace with a rapidly changing future and assure 
improved educational outcomes. 

The National Educational Research Policy and Priorities Board (the “Board”) has among 
its important responsibilities assigned by the Congress the identification of the needs and 
opportunities for educational research and development on a national basis. The Board considers 
the challenges identified above to constitute issues of crucial importance. Furthermore, the Board 
believes that research and development can make an important contribution in framing solutions 
to these problems. For that reason, the Board has undertaken an assessment of the current 
educational research, development and dissemination system, developed a vision of what it 
should be, and described a strategy of how that vision can be made a reality. In this work, it has 
conducted several workshops, commissioned four background papers, and consulted with experts 
and stakeholders. It has made this assessment with an emphasis on candor and realism, because 
of its conviction about the importance of putting research and development to effective use. Now 
is an important time for some basic decision-making about the national education research and 
development effort. 
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This assessment is part of a larger set of activities, all designed to strengthen the national 
knowledge-building system. These include steps to define an acceptable peer review process and 
strengthen the methodological base and standards of evidence for research and development 
work. After an initial review of the research agenda, the Board has proceeded with this 
assessment and a parallel study being undertaken by the National Academy of Education (NAE) 
to define cutting-edge research opportunities and priorities. This body of work is designed to 
invigorate and strengthen educational research and development and its communication and use 
in education problem solving and improvement. 

ASSESSMENT OF THE CURRENT SYSTEM 

Prior to discussing our assessment, the current system of research, development and 
dissemination for education first needs basic description and definition. The Board’s formal 
responsibilities include concern for national efforts, not just the undertakings of the Office of 
Educational Research and Improvement (OERI) and the U.S. Department of Education. This 
assessment is accordingly national in scope, though its special focus is on the Department and 
OERI. 



The language historically used to describe knowledge building and knowledge 
transmission no longer works to convey precise meaning in the education field. Basic versus 
applied research and research versus development are terms which no longer reflect realities 
with clarity. The lines between applied research, technical assistance and dissemination blur 
when one is dealing with the practical problem solving and information needs of a 
comprehensive reform effort in a local school district. As we better understand the interactive 
nature of knowledge building and the important role of school communities in the process, the 
linear model from research to implementation no longer captures an ideal process. Yet this 
existing language is a vital bridge to a more refined understanding. It is important to note at the 
outset that the Board perceives research and development (R&D) as a continuum of activity from 
fundamental understanding of phenomena and behavior to applied research, to initial 
development efforts to substantial trials or demonstration of concepts and systems which hold 
promise based on research and initial development. 
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The meaning of the terminology employed is important to the assessment of the current 
system and even more important to the implications of that assessment. All R&D is not the same 
in purpose, scope and methodological approach. Fundamental research is generally aimed at 
understanding basic phenomenology or behavior about important topics or poorly understood 
problems, not necessarily explicitly connected in its content to application or practice. Applied 
research generally takes new research findings into applied settings to test their relevance and 
impact in small ways. The more promising results of such basic and applied research are then 
taken into increasingly larger settings. This should be done under sufficiently rigorous conditions 
to develop, demonstrate and assess outcomes with confidence before moving the concepts into 
general practice. 

This model of the R&D process is one of long-standing and continued use in many public 
and private R&D organizations. It evolved in the physical sciences, and has, therefore, less 
automatic applicability in the fields of social and behavioral sciences, particularly in the 
distinctions between basic and applied research. Further, not all work can or should proceed 
mechanically in linear fashion. Nonetheless, the model has stood the test of time in many 
domains, and provides for educational R&D a lens through which one can usefully examine the 
characteristics and behavior of the system. Equally important, the system has provided a 
framework through which the research community can communicate and explain the value and 
results of its knowledge-building proposals and activities to the funders and the public. It is 
interesting to note that the U.S. Department of Defense structure for this continuum has not 
changed in any fundamental way since it was first established in the early 1960s. 

Our topic also includes that part of the knowledge-building process, most often labeled 
dissemination, which facilitates and transforms what we know into what we do. Dissemination in 
its traditional use connotes passive sources of electronic or written information and techniques to 
make that information accessible to those who need and want it. Dissemination has evolved from 
a paper process to an electronic one as well, as increasing volumes of information and data — 
some of it unassessed — have flooded the information market. While still vital, passive 
dissemination is increasingly understood to be necessary but not sufficient for the highly diffuse 
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and decentralized educational system, particularly when a state or school district is undertaking 
large scale and intense efforts, such as comprehensive reform. In such cases, more proactive 
processes are needed in addition to traditional dissemination, which include aspects of applied 
research and technical assistance. Further, many programs, including the special education 
program in the U.S. Department of Education, have incorporated the crucial role of the national 
media in penetrating the consciousness of professionals in the field as a simulator to seek more 
information about innovative practice. 

The Board in no way wants to replace traditional dissemination, though continued 
analysis of its cost and effectiveness is appropriate. However, the Board believes that a more 
comprehensive approach needs to be undertaken and conveyed to pursue the effective 
transformation of what we know to what we do. It contains elements of both utilization and 
communication, but we have selected the term “communication” to convey this broader 
meaning. 

Also as a matter of definition and description, some important aspects of a 
comprehensive approach to any credible knowledge-building process will receive less attention 
in this assessment, not because of their lack of .importance, but rather because of the need to 
concentrate on elements most in need of attention and improvement. The first of these 
components is the collection of valid national data. Their importance in education has grown 
more evident as the National Center for Education Statistics (NCES) has become better financed 
and continued to mature as a statistical agency. The growing body of regularly and specially 
collected data is a critical part of sound research and a continuing source of performance 
indicators. Although this report does not dwell on the NCES program, it is not from lack of 
recognition of its continuing importance to the overall system. NCES will need to be a full 
partner in the development of learning agendas, and in making choices among the many 
important issues which might benefit from new or better information. 

A second component given modest attention is the existing national education 
dissemination system known as ERIC (Educational Resources Information Center). That system 
provides an important foundation for the traditional national dissemination system, much used by 
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scholars and practitioners. ERIC has been, and will continue to be, an important actor in the 
broad knowledge communication job, which needs to be done. ERIC has been criticized for the 
absence of a quality screen on the material it contains, but the Board understands that Herculean 
task begins with the establishment of standards of evidence now being developed by the 
Assistant Secretary for OERI. Further, such screens will obviously be implemented gradually. 

Finally, this assessment gives primary attention to elementary and secondary education 
(K-12) rather than to postsecondary or higher education. The federal investment in R&D in 
postsecondary or higher education is relatively small and split among offices and agencies. Much 
of the work is sponsored by foundations and the institutions themselves. Though small, the work 
is important; but it does focus on common and unique problems in a quite different context 
which would require special treatment in this assessment. Given the overwhelming urgency and 
importance of the K-12 educational R&D issues, the Board has chosen to focus there. 

Educational RDC Spending 

It is reasonable to start an assessment with a description of the scope and content of 
educational research, development and communication (RDC) activities and spending levels 
across the nation. That logical beginning is frustratingly impossible from any readily available 
data sources. There are small islands of information about isolated components of the total, but 
the identification of educational activities from other spending purposes, RDC activities from 
identifiable educational spending, and useful sub-categories of RDC activities are too often 
unavailable. 

A background paper commissioned by the Board undertook to provide as much of a 
mapping or profile as the data permit, using estimation where no other sources exist. The total 
national educational spending on education totaled $530 billion (inclusive of elementary, 
secondary and postsecondary) in 1995, some 7.3% of our gross domestic product (GDP). This 
fraction of total GDP is significant, though not as large as health, which represents in the range 
of 13% of GDP. The amount spent on educational RDC is necessarily a guess. If one seeks to 
add up the educational RDC spending in the U.S. Department of Education and among 
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foundations, using generous definitions and interpretations, one can reach annual totals in the 
range of $900 million to $1 billion. It is clear that other federal agencies — the U.S. Departments 
of Defense, Health and Human Services, and the National Science Foundation among them — 
spend significant sums not routinely classified as educational RDC. State and local governments 
and universities, from other than federal and foundation funds, spend more modest amounts as 
well on RDC. Taking these organizations into account, one might postulate that total educational 
RDC may approach $2 billion annually. 

With a $2 billion annual expenditure level of educational RDC activities, the nation 
would be investing less than one half of one percent of the total enterprise in educational 
knowledge building, an amount the National Research Council (NRC) estimated was 30 times 
less than is spent in health R&D. Other kinds of activity of national importance will invest a far 
higher percentage of total resources in knowledge-building activities; thus by any comparative 
measure, the nation is significantly under-invested in educational R&D. 

The weight of expert opinion confirms the comparative analysis. The reviews of 
educational knowledge-building of the early 1 990s by the National Academy of Sciences (NAS) 
and the National Academy of Education (NAE), and the more recent review of technology to 
strengthen K-12 education in the United States by a panel of the President’s Committee of 
Advisors on Science and Technology (PCAST), all found educational R&D insufficiently funded 
on a national basis. The most recent review recommended a $ 1 .5 billion annual increase in 
educational R&D for K-12. Even such a large increase would still leave educational R&D well 
below average in knowledge-building spending relative to other fields. 

The national spending on educational RDC reflects not only a smaller amount than 
warranted by the size and importance of educational activity, but also is distributed in a way that 
reflects little or no sense of strategy and priorities, resulting in a set of gaps and imbalances. 
These undesirable characteristics are, in part, caused and made more difficult to correct by 
organizational arrangements and processes within the broader national RDC system. Indeed, the 
term “system” may imply more rationality than the fragmentation of resources and the lack of 
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intellectual focus warrant. Both the substance of the work undertaken in the system and the 
processes by which it is planned, conducted, and used have been included in this assessment. 

The Strengths of the Current System 

Assessments naturally tend to focus on improvements to be made and problems to be 
fixed, and this one will not be an exception. The Board regards the problems as serious and 
urgent. But the assessment would be remiss in not calling attention to the strengths and 
opportunities which are present, and the progress which has been or is being made. 

The major asset on which to build improvements to the system is the very large, as yet 
unmet, demand for more knowledge and information useful in classrooms that would achieve 
more effective educational practice to meet changing and escalating educational goals in a 
changing demographic, social and technological context. Further, while financial support is 
nowhere near appropriate levels, some resources are available, and the case for more is strong if 
the understanding, justification and performance potential are persuasively argued. There is a 
very broad range of R&D, particularly applied work, now being undertaken from which to select 
and build. 

Prior reviews of this subject by the NAS and NAE in the early 1990s had a variety of 
recommendations, some of which have been actively pursued. For example, the OERI research 
program had been urged to provide for more field-initiated work, and that guidance has been 
substantially implemented in the agendas of the centers and labs. In this regard, it is important to 
distinguish between: (1) the formal field-initiated research program which represents 20% of the 
research budget and solicits proposals directly from researchers, and (2) the amount of specific 
work that is proposed from the bottom-up rather than the top down. In the latter category, a very 
large percentage of the work of the research centers emanates from within the center in response 
to broad areas of interest. Further, the work of the regional laboratories is largely in response to 
field level demands. In this broader sense, a much higher percentage than 20% of specific 
research projects is instigated in the field among researchers. 
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Within the Department of Education, there is much to be learned from the separate 
special education R&D program of the Office of Special Education and Rehabilitative Services 
(OSERS). While not trouble free in its insularity, the program demonstrates the importance of 
infrastructure, particularly the user and beneficiary support structure that stimulates and sustains 
the program and helps shape its objectives. Special education, in its demand profile, stands in 
sharp contrast with most of the rest of education R&D. There are also likely to be important 
lessons in the current experimentation in special education to substitute a more inclusive 
planning process for sharply defined statutory resource allocation formulas. 

Concerns about the research standards and other standards of evidence in educational 
R&D are being addressed, in the first instance by the Board, and for standards of evidence by the 
Assistant Secretary for OERI. Our review and the continued urging of the Congress reinforce the 
need for quality assurance through the use of peer review systems. The recent study of the 
National Institutes of Health (NIH) by the Institutes of Medicine and our background paper make 
clear that peer review is neither foolproof nor the only effective approach to quality assurance, 
but it is one of the best tools we have. It should be extensively used in selecting specific projects. 
But peer review systems are far from the only component of quality assurance and priority 
setting. There are also issues of standards of evidence and methodologies that need serious and 
continuing attention, as this assessment will later discuss. 

The communication system has been improved through the use of technology for better 
access to RDC products for those who seek them out. Much remains to be done with respect to 
coverage, timeliness and quality screens, but the Department appears sensitive to the needs. 

The recent major implementation of the Government Performance and Results Act 
(GPRA) of 1 993 at the federal level, which mandates the development of strategic plans, 
performance measures and annual performance plans, is providing an environment for focusing 
longer range research, development and knowledge-building needs associated with strategic 
objectives and budget resources. It will be important to ensure that policymakers, researchers and 
consumers understand that all research in a sensible and sound program is not instantly 
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convertible into practice and that time horizons must stretch over several years, or in some cases, 
abandoned. But this understanding can be built and does exist in areas such as health. 

The Department is exploring the development of longer range learning agendas, and is 
including R&D support for its strategic objectives. These statutory processes can also be used to 
focus on RDC system objectives. The Congress is making substantial use of GPRA provisions, 
and the recent House Appropriation Committee Report on the Fiscal Year 1999 budget request 
from the Department of Education makes clear its intent to require more specific performance 
measures to support appropriation requests including for research, development and 
communication activities. It is encouraging that the Congress wants to increase the use of 
evidence in making decisions. It is also important to understand how new and developmental the 
performance measurement system is, and particularly, the limitations of existing methodologies 
to assess the impact of small federal financial contributions to large efforts to achieve broad 
educational improvement goals. 

The Issues in the Present System 

This section describes in summary form the major issues that the Board perceives in the 
existing RDC system, organized in terms of adequate resources, balance and linkage, and 
processes. The assets and developments described above are commendable. They are, however, 
insufficient to address both long-standing problems and the increasing need for better returns 
from the investments in the RDC system. 

Adequate Resources. Those long-standing problems start with the insufficient aggregate 
resources in RDC, identified earlier, which in turn cause other problems. Too few resources have 
led to spreading available resources thinly over a large number of topics rather than 
concentrating on fewer issues. The thinly spread resources include funds, staffing and 
organizational units and institutions. This strategy is unworkable. Resources need to be applied 
to major problems and areas of greatest need. The long-standing problems also include the 
perception that educational research is based on ideology rather than “science,” a view that gains 
currency in a resource-starved environment. 
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• The OERI institutes, created in 1992 to refocus education R&D to important 
educational topics and problems, are a prime example. Notwithstanding some 
bright spots, the institutes lack sufficient internal staff to mount credible programs 
to meet their mandates for comprehensive and high quality work and to provide 
national leadership on critical issues. The lack of internal and external resources 
means that the “critical mass” found in other research institutions to be necessary 
for an effective, high quality program is missing. With the current organization 
and resources, the institutes will sink into increasing irrelevance. This concern 
about critical mass extends to the R&D centers, which, in many instances, are far 
too thin for the work and leadership expected of them. 

• In somewhat different, but equally important roles, the regional educational 
laboratories have immense missions, but only modest resources to achieve them. 
Some have addressed the critical mass problem by aggressive efforts to obtain 
resources from other sources, but they all still face a mismatch between ambitious 
missions and limited resources. And beyond the mission and resource mismatch, 
the regional laboratories face the added complexity of diffuse leadership and 
sponsorship from the federal and state local levels. 

• The relationship between the centers and the labs is too distant and troubling. 
There are concerns that in a period of woefully inadequate resources, the nation is 
reaping less than it could from the collective investment in the two enterprises. 
The problem starts with perceived roles. Perhaps in caricature, the centers are 
sometimes seen as the thinkers and the regional laboratories as the interactors 
with practitioners. Any presumed virtue in such a split is out of touch with the 
realities about how practice will improve and how comprehensive reform is being 
implemented effectively. Both sets of institutions need to be thoroughly grounded 
in the realities of practice, while playing complementary and collaborative roles in 
solving critical RDC problems. The centers may emphasize their comparative 
advantage in broad analysis and conceptual framework, while the laboratories 
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may emphasize integration with practitioner settings; but both should function 
closely together to maximize their collective impact. 

• If the quality, utility and resources for educational RDC are to improve, more 
effort, focus, and resources will be needed to strengthen the supporting 
infrastructure in three different respects. First, user and consumer demand for 
better and more useable knowledge should more strongly reflect the underlying 
need. This will require a substantial educational effort and a strategy of inclusion 
in planning and execution. This effort will necessarily encompass better education 
of teachers to the value and use of R&D. Second, more resources can be used to 
attract new, highly qualified scholars from many disciplines to educational R&D 
and its issues. Lack of resources and prestige now inhibit such recruiting. Third, 
the institutions that undertake the critical work will need to be selected on merit, 
nurtured with sustaining resources and the demand for quality, and rigorously 
evaluated for performance. 

• Up to now, there has been no systematic effort to trim the mission expectations of 
the federally financed RDC institutions to fit the limited resource availability. If 
more resources were to be forthcoming, this imbalance would start to correct 
itself. If, however, the expectations of more resources are unrealized, continuation 
of the mission-resource mismatch will have corrosive and cumulative 
consequences. A cycle of unfulfilled expectations leads to falling confidence, 
smaller resources, diminished capacity to recruit talent, few important results and 
reduced de facto expectations. This spiral will produce a series of worst outcomes 
for funders, researchers and the practitioner co mm u ni ties; and the institutes, in 
particular, are rapidly heading in that direction. Mission must be matched with 
money. 

• Closely related to the matching of mission and resources is the issue of priorities 
and focus. The attention to both has been more rhetorical than real. The many 
topics of interest in educational knowledge building tend to be treated as if they 
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were of equal importance. A more serious job of priority setting is needed; and to 
that end, the Board has a parallel effort now underway on priorities. Those 
priorities need to be set with the educational problems to be solved firmly in 
mind. 

Balance and Linkage. The issues of inadequate aggregate resources, critical mass and 
absence of priorities are coupled with serious issues concerning the balance and linkage in the 
continuum of educational RDC activities. This continuum ranges from fundamental research 
through large-scale demonstration and effective communication of knowledge and information to 
the practitioner community. The issues include: 

• The portfolio of the U.S. Department of Education is bunched in two areas along 
this continuum: (1) applied research and small scale development; and (2) com- 
munication activities. The Department conducts essentially no basic research, and 
is not deeply involved in large-scale development or demonstration, especially 
about comprehensive or standards-based reform. The fundamental research — 
largely in the cognitive and neural sciences — is conducted elsewhere in other 
federal agencies, most notably the National Institute for Child Health and Human 
Development (NICHD), the Office of Naval Research (ONR), and to a lesser 
degree the National Science Foundation (NSF). There is also modest foundation 
support for basic educational research. The OERI institutes are contributing some 
of the important applied research aimed at comprehensive or standards based 
reform, testing and assessment. Altogether, it is not possible to characterize the 
total fundamental research programs as robust. 

• Large-scale development, particularly centered on major comprehensive reform 
efforts, is supported by foundations, the Department and other private resources. 
The Department’s participation has generally been modest and evaluative. It 
appears to have had some of its best success when demonstrations are tied to 
formula grants. While foundation and private sponsorship should be welcome, the 
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Department’s direct sponsorship or other participation should be more vigorous 
than it is, provided the resources are made available for the purpose. 

Rather than complementing the focus of the Department, the pattern of foundation 
funding, with the exception of the few foundations supporting large-scale 
comprehensive reform experimentation, mirrors the federal focus. Foundations do 
appear to give more focus to curriculum and teaching topics than the federal 
program. The overlap may be healthy, but gaps are not. It is, moreover, the 
Board’s impression that the fit and relationship are coincidental rather than 
deliberate. 

The location and level of fundamental research are of concern in several different 
ways. It is not desirable that basic research be sponsored or conducted in one 
organizational framework, but it is important that such work be linked to the 
applied research and ultimate practitioner communities that will make use of its 
findings. Staff in applied research organizations who understand the findings and 
who design the applied research will make the necessary linkages. And the 
linkage requires the identification of application problems from practitioner and 
applied research communities. Efforts to link across organizations are occurring 
more frequently, such as recent planning work concerning a new initiative among 
OERI, NICHD and NSF, but nowhere near the extent needed. 

The absence of substantial large-scale development activity with rigorous 
research and evaluation aimed at critical problems is noticeable. This concern is 
particularly strong in light of the continuing difficulties of scaling up small, 
promising developments that require systemic change for widespread success. 
Evidence of effectiveness on a large scale provides protection from faddism and 
insufficiently tested ideas. 

In those areas where demonstration activity does exist, the Department is making 
increasingly strong efforts to insure a knowledge-building component in the 
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programs or projects in the form of research or evaluations. These activities are 
undertaken not only in the Office of Reform Assistance and Dissemination 
(ORAD), but also throughout the Department’s programmatic offices. This 
dispersion of demonstration activity to the organizations with primary interest in 
the substance of the work is appropriate; and growing attention given by the 
Department and the Congress to the knowledge building component is welcome, 
and should be continued. What is not in place is a sense of strategy through which 
demonstration activity is linked to high priority problems and targets of 
opportunity that solid research suggests should be exploited through more 
widespread demonstration activity. 

At a more general level, concerning the gaps and unevenness, the science base 
supporting the applied agenda of the OERI and Departmental R&D activities is 
not clearly visible, which adversely affects its force and credibility. In some areas 
the science base does not exist at all. The existence and visibility of that base is an 
important element of its persuasiveness to funders. Such persuasiveness will 
require clear standards, more rigor, and a larger, better-integrated learning agenda. 

The proper involvement of the practitioners in knowledge-building and 
implementation activities is stimulating new thinking in many areas, including 
education. It is increasingly clear that practitioner success and acceptance of 
revised practice is strengthened by understanding and involvement. This 
realization is leading toward efforts to seek active participation and linkage of 
operating organizations and their teachers in the R&D planning, conduct and 
evaluation process by moving them to become “learning organizations,” a concept 
with potential for schools. In this role, the practitioner community becomes vested 
in the objectives of the innovation and reform, provides helpful input to fitting 
concept to operational reality, and contributes a continuing basis for 

% 

accountability and mid-course correction. Other supplementary ways are needed 
to create incentives for better acceptance of proven practice improvements. 



• The more traditional approaches to dissemination, which leave to those in need of 
exemplary practice and sound knowledge the burden of finding it, are being found 
inadequate. This is particularly true for those undertaking large-scale 
comprehensive reforms. Even with the Internet and other forms of electronic 
access, the passive systems don’t fully meet the needs of those with ambitious 
innovation agendas, and the volume of information can overwhelm the 
practitioners. The more intensive efforts appear to require a combination of 
traditional dissemination, technical assistance and short-term applied research or 
problem solving. A new set of intermediaries and adjustments in existing 
organizations are emerging to meet these needs, and dissemination needs to be 
reconceptualized in this broader context. It has been labeled here as 
communication to distinguish it from too narrow an approach. Seen in this light, 
the communication activities concerning the transfer of needed knowledge and 
information are found in many places. Communication activities are widely 
scattered in OERI and elsewhere in the Department. The technical assistance 
operations are housed in the programmatic components and the staff agencies of 
the Secretary’s office. While consolidation of these different operations is not an 
attractive option for many reasons, a process which defines roles and links these 
related activities warrants more attention as user needs change. 

Processes. The Board’s review of the current RDC system included specific attention to 
important processes by which agendas are set, support is mobilized, resources are allocated, and 
progress is made, assessed and made known. The experiences and models in other federal R&D 
programs were examined for applicable lessons. These comparative systems are, of course, 
molded by the nature of the activity and its context, as is the case for education, thus making 
mechanical replication unattainable. Nonetheless, comparisons can provide insights for 
assessment, and four seem particularly salient to the current issues in the educational RDC 
process. Each involves matters in which there are important tensions, requiring a careful balance. 

• The first involves the issue of agenda setting, in which two important objectives 
should be harmonized. On the one hand, long experience suggests that R&D is 
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most productive if the researchers are given substantial latitude in initiating work 
on their own ideas or perceived problems. This is generally known as field- 
initiated research. On the other hand, there is a strong and continuing need to give 
the R&D agenda a sense of national focus and priorities aimed at the most 
important gaps in our knowledge and the most promising research approaches. 
The point is not to swing back and forth between these two important objectives, 
as has sometimes been the case in education, but to find ways to achieve them 
both in a balanced way. OERI has adopted several of the devices to create the 
desired harmony. The field-initiated emphasis for particular projects is well 
ingrained, complemented by a set of national issues and research questions within 
which the specific proposals can be reviewed. The national focus part of the 
balance, however, remains weaker than it should be. 

One mechanism employed in NIH to help strengthen the national focus is a set of 
intermediate councils of external expertise from the research and practitioner 
communities that continues to work on the national priorities by building broader 
scientific judgment about important issues which leads to consensus about 
research needs and findings. Education lacks a similar process that ensures a 
continuing dialogue among OERI, the institutes, centers and regional laboratories 
plus other important educational research institutions and sponsors about critical 
research problems and opportunities. Filling this gap can be an important 
milestone in developing consensus about national priorities. 



The second issue centers on mobilization of support, discussed in part above in 
connection with adequate resources. Given the importance of federal funds in 
educational R&D, mobilization of support with respect to processes means the 
involvement of the federal political process in the RDC system. This interaction 
has had a troubled history, leading to suspicion and hostility. Continuation of 
adversarial attitudes is detrimental to the development of an effective educational 
RDC system. Respect for the responsibilities of the appropriate domain and of the 
research community and political policymakers is both necessary and a reality in 
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other federal R&D programs. Where it works well, all parties — policymakers, 
researchers, operating officials and consumers — are engaged in setting broad 
objectives and parameters of the work, while leaving specific design and 
execution of projects to the sponsoring agency and the research and practitioner 
communities. At the present time, more effort appears to be directed at protecting 
the parties from each other rather than in collaborative objective setting. 

It should be clearly recognized that this suggestion does not and should not put 
the Congress and its committees into the detailed design of specific projects, 
organizations, and processes. There has been far too much of that to the detriment 
of educational R&D quality, objectivity and productivity. At the same time, it 
should be equally clear that the Congress and its committees, as the sponsors and 
overseers of the work of the Department, have a crucial role in the establishment 
of goals and objectives and the oversight of accountability for their performance. 
It is not beyond the capacity of the parties to put in place collaborative processes 
for fulfilling those roles. 

• The third issue centers on the allocation of resources, but is closely related to the 
roles of the Congress and the Department discussed above. The allocation of 
appropriated resources in OERI, particularly to and by the institutes, is heavily 
controlled by statutory and other distribution rules. While this rigid allocation was 
adopted to meet a particular concern at a prior point of time, its continuation 
frustrates responsiveness to new needs and circumstances; and it can be 
particularly damaging in dealing with new national priorities. In a well 
functioning system, these structures are simply unnecessary. For example, the 
emphasis on field initiated work is a strong objective in NIH, and is being 
achieved without statutory or regulatory prescription. Further, the Congress is 
experimenting with more discretion within the special education R&D structure. 
Given assurances among the parties, such an increase in flexibility should be tried 
in OERI. 



O 

ERIC 



20 22 



• A fourth issue is the development of appropriate research methodologies and the 
establishment of standards of evidence to be applied to knowledge-building 
activities. The question of methodologies is a long-standing and difficult one. 
More rigor is desired and needed. Yet traditional rigorous methodologies such as 
randomized experiments are not routinely feasible, affordable or appropriate in 
education research, though control group methodology has a continuing and 
important role in the clarification of critical research issues. Further, the good 
work done by NICHD in reading research should not be regarded as an all- 
purpose endorsement of the medical research model. The output of educational 
R&D must be fitted to school classrooms where other approaches and measures 
will be required to assess impact and outcomes. 

The Board believes that endorsement of rigor needs to go further in detailing 
appropriate techniques. The choice is not between randomized techniques and 
nothing at all. There is an appropriate level of rigor associated with the stage and 
purpose of the research being undertaken. Carefully constructed case study work 
may be thoroughly appropriate for “micro” applied research to construct reliable 
hypotheses for further testing, but is clearly insufficient for confident projection 
of success in nationwide application. The Assistant Secretary for OERI and the 
Board are already engaged in the development of appropriate standards of 
evidence and more rigorous methodological standards, and will continue 
vigorously to pursue these important problems. 

As we reflect upon the assessment of the current system, it is our conviction that 
important changes are needed if the educational RDC system is to survive and become an 
effective instrument in support of educational progress. Left where it is, the federal part of the 
system is likely to atrophy. That outcome, however, should be avoided, though this cannot 
happen overnight. The building effort will require vision and effort, and should begin soon. The 
important first step is to agree what an effective educational RDC system should be and outline a 
strategy to reach that vision. 
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A STRATEGY FOR IMPROVEMENT 



Attaining the vision of an ideal system when reality is well short of that ideal requires a 
strategy for improvement that can be achieved over a reasonable time period. One must begin 
with incremental steps, but with a clear, planned, and measurable strategy to identifiable ends. 
The Board has no doubt, for example, that the Panel on Educational Technology of the 
President’s Committee of Advisors on Science and Technology set forth a reasonable goal to 
increase RDC federal spending $1.5 billion annually. However, that amount obviously could not 
be wisely spent next year given the existing infrastructure. It is not, however, unrealistic to 
recommend that educational R&D increase to $500 million annually over the next five years. 
Such an increase could be wisely planned and spent. 

The strategy should proceed on two different levels, led by an initiative that charts the 
path for a better, more inclusive and understandable approach to conducting the educational 
RDC system and that undertakes an important knowledge-building task in teaching and learning. 
Such an initiative would contribute to critically needed knowledge in the education of children, 
and serve as a prelude and beacon for further rethinking and action in strengthening the entirety 
of the educational RDC system, particularly within the U.S. Department of Education. This 
across-the-board re-examination and improvement would constitute the second level. Some of 
the key components of both levels are now apparent, while other components will emerge as the 
work proceeds. 

It is possible now to do the planning for such additional resources by adding a learning 
agenda developed through the collaboration of this Board and the Assistant Secretary for OERI 
and linked to the Department of Education’s strategic planning in the GPRA process and the 
Departmental budget. The first steps can be taken to open up the planning to the research and 
practitioner community. It can also include exploration with the Congressional committees about 
objectives and goals, and the integration of learning agendas in the GPRA process. The learning 
agenda, once reviewed and accepted, should be seen as a long-term contract among the 
stakeholders as to its objectives, resources and expectations. That process can grow and deepen, 
but it can be started now. 



The Department and the Administration recommended an additional $50 million in the 
OERI budget and a counterpart $25 million in the NSF budget for a problem-centered 
educational research initiative. While not successful with the Congress, the concept of a new 
initiative in OERI research is sound. The Department and the Administration should renew the 
request for at least as much money with the new Congress. A useful way to begin a new, more 
balanced approach is an example or demonstration in which all stakeholders can gain confidence 
that such an approach can be effectively planned and managed, free of some of the impediments 
identified in the current assessment, while still adhering to agreed plans and principles. The 
Board would clearly focus such an initiative on the most important issue of teaching and 
learning, including applied research, initial development and substantial demonstration. 

A strategy at the second level for balancing missions and resources among the 
educational R&D institutions — institutes, centers, and regional labs — is a difficult challenge. On 
the one hand, one is hesitant to diminish or destroy institutional capacity, if we can get the 
collective enterprise off a starvation diet. On the other hand, it makes no sense in the longer run 
to retain the full institutional structure at its current sub-optimum funding and staffing. Further, 

the combined resource, organization and allocation constraints now imposed on the institutes, for 

/ 

example, makes it impossible for them to fulfill their missions and the educational research 
needs of the nation. Yet the Board believes that components of their R&D work are both 
important and useful. The Board would recommend that time be provided to determine whether 
more resources are forthcoming and whether the results of a demonstration initiative prove 
satisfying. 

In the meantime, the Department should review its processes and structure in two 
different ways. The strategic planning exercises developed for the first initiative and its 
successors about other topics can be used and reviewed as a means of coordinating and 
strengthening the assignments and work of the institutes, centers, and labs as an alternative 
approach to further detailing and prescription of missions. It seems plausible that this more 
flexible tool will be both more productive and satisfying to the participants. In addition, the 
Department should review mission assignments to institutes, centers, and labs that are receiving 
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little or no attention for lack of resources and reduce the scopes of their missions or, where 
necessary, urge the Congress to do so. 

The Department must also find an effective way to increase the internal capacity to 
manage increased effort as well as larger budgets. Without more resources in the next 3-5 years, 
the painful work of reducing or dismantling the scope and number of units in the institutional 
structure should commence. Any reductions that turn out to be necessary should be done with a 
steady eye on the priorities described earlier and an understanding that OERI’s critical niche is 
high quality applied research that can make a difference in school classrooms. 

There is an overarching and critical need to increase the high quality and cumulative 
scientific base for educational R&D. We must enhance the standards and methodologies through 
which educational R&D findings are produced and admit our ignorance of poorly supported 
recommendations. There is too much at stake in the education of American children for the next 
century to do any less. The Board and the Assistant Secretary for OERI are developing the 
standards to which all Departmental work should adhere. In addition, OERI staff should assure 
their knowledge and competence to assess important fundamental research done elsewhere. 

OERI and Department should be playing a more active role in large-scale 
demonstrations, particularly related to comprehensive reform. It should also be recognized that 
more resources will be required for such purposes, and opportunities to fund innovation with 
formula grants should be exploited. The issues of scaling up are too important to leave out of the 
RDC agenda. Further, the full range of federal and non-federal efforts should be viewed 
comprehensively and formally or informally coordinated. 

The development of an appropriate infrastructure is a multi-year undertaking that will 
require resources as well as planning for institution building and the attraction of new, highly 
skilled researchers. The Department should, however, begin soon a coordinated effort to build a 
support base for the program in the practitioner community, including the development and 
communication of model programs for increasing teacher awareness and use of high quality 
research. 
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This report has spoken to the emerging needs for more comprehensive support in 
technical assistance, dissemination and applied research functions especially for those who are 
undertaking comprehensive reform efforts. This report has also called attention to the creation of 
learning communities through which schools, school districts or whole states become actively 
involved from learning agendas to implementation of exemplary practice in the RDC enterprise. 
The Board believes that the OERI structure can play a constructive role in nurturing these 
developments. The kinds of work the regional labs already have underway in combinations of 
applied research, technical assistance and information sharing could well be more strongly 
focused to help states and school districts with comprehensive reform agendas. All of the OERI 
institutions should help create opportunities for the establishment of learning communities with 
local partners for projects from micro-experimentation to large scale development. 

These key elements of the early stages of the strategy can evolve and grow as the effort 
gains momentum, and all should be anchored to the urgent national problem ahead. The time has 
come for change and action. With such change and the momentum it will generate, the national 
educational research, development and communication activities will become an increasingly 
important contributor to vitally needed educational system improvement. Without it, the nation is 
more likely to stumble along unhappy with school performance. We can and should surely do 
better than that. 
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THE EDUCATIONAL RESEARCH, DEVELOPMENT 
AND DISSEMINATION SYSTEM 



AN ANALYTIC MAPPING 
I. A Summary of Distinguishing Characteristics 

A. Background and Introduction 

The educational research, development and dissemination (RD&D) system has been a 
frequently revisited topic over an extended period, dating back 30 years. The issues raised about 
the system remain uncomfortably familiar, no doubt to the frustration of those concerned with 
the collective enterprise and its impact on educational processes and performance. In many, but 
not all cases, the system characteristics and the issues they raise discussed herein will plow no 
new ground, raising questions as to whether the durability of identified problems lies with their 
inherent intractability, a weakness in process or imagination, or a problem of attitudes and a 
failure of will. 

The broader issues in the current educational context can be seen as different or more 
acute than they have been over the history of the educational RD&D system. Those context 
changes point toward desirable change in the way our educational system works, suggesting 
value to be gained from well-done research and dissemination. The basic outlines of the change 
are well covered in scholarly and more popular journals, but not as yet as fully grasped by the 
public. A particularly readable, cogent and recent articulation of the broader context by Richard 
Mumane and Frank Levy (Mumane, R.J. and Levy, F. with forwards by Thomas W. Payzant and 
Robert W. Galvin/Edward W. Bales, 1996) makes the basic points: 

• The cost of competitiveness began in the 1980s to force radical change in the 
skills required to succeed in the economy. In 1979, a 30 year old man with a U.S. 
high school diploma earned an average $27,700 in 1993 dollars. In 1983, that 
man of 30 years earned an average of $23,000 in 1993 dollars; while in 1993, the 
wage for such an individual dropped to $20,000. Only one half of men of 30 
years of age in 1993 had gone beyond high school (a stable graduation rate over 
an extended period). 

• Standard test scores are modestly up from 1980; however, almost half of all 17- 
year olds cannot read or do math well enough to get a job in a modem automobile 
plant (NAEP scores relative to requirements of Diamond-Star and Honda). 

• The earnings differential between college graduates and high school graduates is 
growing, but analysis suggests that the skills required by employers do not neces- 
sarily require college. Those skills include not only the hard skills of reading, 
math and problem solving, but also “soft” skills in teamwork, communication and 
computer utilization. While the demand for college-provided skills has increased, 



college graduates are more likely to also possess those skills which should be 
learned in secondary school. 

• Public opinion, until recently, did not associate educational performance problems 
with their local school and the linkage to changes in the economy, but more recent 
polling indicates a change in views. (See chapters 1 and 2, Mumane, R.J. and 
Levy, F., 1996). 

• This economic context is only part of a substantial demographic and social change 
which further challenges the capacity of the educational system to meet society’s 
needs. 

This broader context creates both problems and opportunities for change in educational 
practice and the research which can fuel it. On the one hand, the public is striking out at the 
status quo in many areas, including the school systems which find themselves under siege in 
many jurisdictions; and on the other hand, the potential market for innovation, research and solid 
information should be high indeed. 

It is in this context as well as the history of educational RD&D that the mapping of the 
existing system begins. 

B. Distinguishing Characteristics 

In order to set the framework for the old and new issues about the RD&D system, it is 
useful to review some of the distinguishing characteristics of the system. We seek to do so 
through the answers to a series of rhetorical questions, some of which are based on clear data, 
others on estimates, and some on opinions of experts and observers. All of the questions and 
their answers influence current and future trends and directions of the system. It is probably 
appropriate to note at this early point that “system” is used as a descriptive reference to how 
intents, resources and institutions concerned with educational knowledge-building have inter- 
acted, and does not imply any comprehensive or orderly process through which actions are taken 
or events occur. 

1. What are the Distributions of Effort and Resources? 

a. RD&D Spending as a Percent of Total Education Spending 

Educational spending in the United States approximated $530 billion in 1995 or 7.3% of 
our gross domestic product (GDP). The aggregate national expenditures on educational RD&D 
are difficult to define and collect. If, however, one combines US Department of Education 
spending and foundation spending (both generously defined), one gets a total volume in the 
range of $0.9-1 .0 billion annually, or about 0.2 percent of total educational spending. This 
excludes educational spending in other federal agencies, state and local governmental agencies 
and institutions, and private colleges and universities with their own funds. Even if these 
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excluded resources doubled the educational RD&D system national expenditures, RD&D expen- 
ditures would still amount to less than one half of one percent of total educational spending. 

Educational RD&D expenditures are a much smaller percentage of total expenditures 
than in other fields such as industry, health care, and defense (Vinovskis, 1996). The NRC staff 
estimated federal health research alone at 30 times larger than education (NRC, 1995), with 
health representing 12-13% of GDP. Citing a 1988 GAO report, Hawley (1990) asserts that 
education research funding is significantly lower than it was 1 5-20 years ago relative to 
education expenditures overall. The federal government spends less on education research today 
than it did twenty years ago in real dollars (Vinovskis, 1996). Biddle (1996) argues that while 
the Federal government pays fewer than 10 percent of school operation costs, only one tenth of 
one percent of those funds are allocated to research. 

b. Basic Research, Applied Research, and Dissemination 

According to Hawley (1990), the share of research funds allocated to basic and applied 
research has declined because of a desire by policymakers and practitioners for OERI and other 
ED research programs to resolve problems. Within OERI, resources have been diverted from 
research to dissemination. Hawley cites a 1989 Department of Education budget which shows 
that over 90 percent of OERI’s research and improvement budget is spent on institutional 
support. This includes university-based research centers, the Educational Research and Infor- 
mation Clearinghouses (ERICs), and the regional education laboratories. He asserts that this 
focus on institutions restricts ED’s support to a limited number of topics and hampers researchers 
from focusing on new problems in education research. 

Other critiques of the educational RD&D system call attention to distributional character- 
istics of the system which reinforce or extend those noted above including: 

• A limited amount of effort devoted to basic research relative to applied research, 
and more attention devoted to dissemination than to research. 

• Small rather than large applied projects initiated by federal sources. 

• A dissemination system that may be relatively comprehensive, but is also reactive 
(rather than proactive), inefficient and not discriminating. 

2. Who Spends the Resources on What? 

The educational RD&D investments of the sponsors of such work are as difficult to 
identify precisely and comprehensively as the distributions discussed in the prior section. The 
federal government accounts for the largest share of RD&D spending, which is done by the US 
Department of Education and other federal agencies, and is in the range of 50-100% more than 
the next largest sponsors, the foundations (a rough estimate based on available data). The 
spending by state and local governments (including state and local govemmentally-owned 
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educational institutions) plus private postsecondary institutions make an additional, but relatively 
small contribution. An unknown, but likely small contribution to educational RD&D is also 
made by private suppliers of educational materials. 

Federal 

The federal expenditures for educational RD&D are led by the US Department of Educa- 
tion. In response to a Congressional inquiry on research funding in FYs 1995 and 1996, the 
Department identified somewhat more than $600 million for FY1995 (includes a liberal view of 
RD&D, but excludes NCES). These figures are contained in Table A-l in the Data Attachment. 

Hawley (1990) estimated that OERI was responsible for one sixth of ED’s research and 
development funds; and the OERI portion of the total program reported to the Congress for FY 
1996 was a comparable 16%. As inspection of Table A-l makes clear, the large bulk of the work 
is applied research and dissemination activity aimed at particular programs, plus evaluation 
activity. With respect to the character of this federal activity which we will address in specific 
ways throughout this paper, it is interesting to note not only its diversity, but also areas of limited 
activity. 

There is relatively limited evidence of substantial basic research work, though the classi- 
fication systems of the data do not facilitate the identification of such work. Beyond the 
evaluation of charter schools, the mounting of large experimental projects and scaling up of 
reforms and innovative small projects are not receiving federal sponsorship. As noted in the 
RD&D System Committee’s September workshop, most school reform efforts taking place in 
schools today have not been funded by the federal government. Instead, these school reforms 
have received funding from foundations and corporations for development, evaluation, and 
funding (Hawley, 1990). Additionally, schools often contribute to dissemination funding 
through their Title I funds (Slavin, 1997a). 

Hawley (1990) cites the numerous federal agencies involved in funding education 
research. These include: 

• National Science Foundation 

• National Institute for Child Health and Human Development, Department of 
Health and Human Services 

• National Institute on Disability and Rehabilitation Research, Department of 
Health and Human Services 

• Department of Labor # 

• Department of Defense 

The federal budget documents prepared by OMB do not readily provide the information 
on the amounts and content spent by these other agencies on educational RD&D. Such informa- 
tion, if available at all, must be obtained directly from sponsoring offices. As a general matter, 
most of the educational RD&D in agencies other than the US Department of Education (ED) is 



directed to work directly supportive of the agencies’ mission. It also tends to be applied. That is 
not necessarily the case, however, for either the National Science Foundation (NSF) or the 
Department of Defense (DOD). NSF is explicitly interested in basic research; and the DOD’s 
broad interests led it into considerable basic and applied work related to education, training and 
human behavior as it relates to its large training and operational activities. 

Office of Education Research and Improvement (OERI) 

Fiscal year 1997 funding was $398.1 million. 47.6 percent of this was for the traditional 
programs - NCES, the centers and labs, field-initiated research, and ERIC. Increasing amounts 
are spent on educational technology and dissemination. In FY1997, approximately the same 
amount was spent by OERI on education technology projects as on traditional centers, labs, field- 
initiate research, and ERIC combined (Vinovskis, 1997). 

OERI declined in staff members from 448 in FY92 to 358 in FY96, and to 338 in FY97. 
At the same time, the real budget of OERI has increased, so that each employee represents $1.5 
million in constant 1996 dollars (Vinovskis, 1997). 

Within OERI, the primary elements include: the National Research Institutes with their 
components — university-based education research centers, field-initiated research and directed 
programs; the Office of Reform Assistance and Dissemination with their components — develop- 
ment and demonstration programs, state and local support including the regional laboratories, 
learning technologies initiatives, professional development, recognition and dissemination 
programs; and dissemination activities embedded in several OERI organizations. The resource 
levels for these activities are indicated below. 

National Research Institutes. The fiscal year 1997 funding priorities are shown in 
Table 1 below. The five institutes were authorized at $100 million for fiscal year 1996 
(Vinovskis, 1997). However, the actual amount available was only $43 million in FY96 and 
FY97. Institutes must spend at least 20 percent of their budgets on field-initiated research in 
FY96 and FY97, and 25 percent in 1998 and 1999. At least one third of the funds were to be 
allocated to national research centers in FY96. Table 2 shows the distribution of FY1996 
resources by institute, taking account of the various mandates and set-asides. 
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Table 1 

NATIONAL RESEARCH INSTITUTES FUNDING LEVELS 

(in thousands) 



1995 



1996 



1997 



Achievement Institute 
Centers 

Field-initiated studies 
Special studies 
Cross-cutting activities 
Peer review 



$13,700 

0 

0 

0 

0 



$9,600 

2,580 

0 

600 

120 



$9,600 

2,580 

100 

600 

20 



Subtotal, Achievement Institute 



13,700 



12,900 



12,900 



At-Risk Institute 
Centers 

Field-initiated studies 
Special studies 
Cross-cutting activities 
Peer review 



7,600 

0 

0 

0 

0 



9,000 

2,580 

0 

1,200 

120 



9,000 

2,580 

120 

1,200 

0 



Subtotal, At-Risk Institute 



7,600 



12,900 



12,900 



Policy Institute 
Centers 

Field-initiated studies 
Special studies 
Cross-cutting activities 
Peer review 



4,800 

0 

3,200 

0 

0 



2,800 

860 

0 

600 

40 



2,800 

860 

40 

600 

0 



Subtotal, Policy Institute 



8,000 



4,300 



4,300 



Early Childhood Institute 
Centers 

Field-initiated studies 
Special studies 
Cross-cutting activities 
Peer review 



1,500 

0 

0 

0 

0 



2,750 

1,290 

500 

1,850 

60 



2,750 

1,290 

560 

1,850 

0 



Subtotal, Early Childhood Institute 



1,500 



6,450 



6,450 



Postsecondary Institute 
Centers 

Field-initiated studies 
Special studies 
Cross-cutting activities 
Peer review 



5,400 

0 

0 

0 

0 



5,000 

1,290 

100 

0 

60 



5,000 

1,290 

160 

0 

0 



Subtotal, Postsecondary Institute 



5,400 



6,450 



6,450 



TOTAL, Institutes 



36,200 



43,000 



43,000 



(Non-add) 

Total, Centers 
Total, Field-initiated studies 
Total, Special studies 
Total, Cross-cutting activities 
Total, Peer review 



33,000 

0 

3,200 

0 

0 



29,150 

8,600 

600 

4,250 

400 



29,150 

8,600 

980 

4,250 

20 



Source: US Department of Education, Justifications of Appropriation Estimates to the Congress, 1997. 



The implications of the reduced internal staffing resources for OERI on the institutes are 
shown in the current distribution of institute staff on Table 3. 
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NATIONAL INSTITUTE FUNDING FOR FISCAL YEAR 1996 



Institute 


(a) 

Funds to 
Support 
Activities 
Contract (10% 
of total) 


.(b) 

Funds to Board 
(2% of 
$1 million) 


(C) 

Funds for Field 
Initiated 
Research (20% 
minimum) 


(d) 

Funds to 
Support RC(s) 
,(33% 
minimum) 


• (e) 

Funds 

Available for 
Any Type of 
Activity by the 
Institute 
(maximum) 


Total 
(FY 96) 


At-Risk 


3,000,000 


225,000 


5,355,000 


8,925,000 


12,495,000 


30,000,000 


Achievement 


3,000,000 


225,000 


5,355,000 


8,925,000 


12,495,000 


30,000,000 


Policy 


1,000,000 


75,000 


1,785,000 


2,975,000 


4,165,000 


10,000,000 


Early Childhood 


1,500,000 


1 12,500 


2,677,500 


4,462,500 


6,247,500 


15,000,000 


Postsecondary 


1,500,000 


1 12,500 


2,677,500 


4,462,500 


6,247,500 


15,000,000 


Total 


10,000,000 


750,000 


17,850,000 


29,750,000 


41,650,000 


100,000,000 



Source: Pelavin 



Table 3 

1997 STAFF LEVELS 



Institute 


1997 Staff 


Professional 


Total 


Education of At-Risk Students 


13 


17 


Student Achievement, Curriculum, and Assessment 


18 


22 


Educational Governance, Finance, Policy Making, 
and Management 


9 


14 


Early Childhood 


6 


10 


Postsecondary, Libraries, and Lifelong Learning 


11 


14 


Totals 


57 


77 



Source: OERI data, 1997. 



National Education Research Centers. The centers have had a changing level over the 
years, rising to as many as 1 8 with quite small budgets. More recently, the centers have been a 
smaller number, and stand at twelve in FY97. The average annual spending of the centers was 
$2.76 million in FY97. However, only the National Center for Research on the Education of 
Students Placed at Risk ($5 million) and the National Center for Research on Education, 
Diversity, and Excellence ($4 million) are well-funded. Eight other centers are funded at $2.5 
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million to $2.8 million each, and another at only $1.5 million (Vinovskis, 1997). The new 
centers often have partners at many colleges and universities, making coordination very complex. 
The current national education research centers are: 

• Center for the Improvement of Early Reading Achievement (CIERA) (University 
of Michigan) 

• Center for Research on Education, Diversity and Excellence (CREDE) (Univer- 
sity of California, Santa Cruz) 

• Center for Research on the Education of Students Placed At Risk (CRESPAR) 
(Johns Hopkins University) 

• Center for Research on Evaluation, Standards and Student Testing (CRESST) 
(University of California, Los Angeles) 

• Center for the Study of Teaching and Policy (CTP) (University of Washington) 

• National Center for Early Development and Learning (University of North 
Carolina-Chapel Hill) 

• National Center for Improving Student Learning and Achievement in Mathe- 
matics and Science (University of Wisconsin) 

• National Center on Increasing the Effectiveness of State and Local Education 
Reform Efforts (University of Pennsylvania) 

• National Center for Postsecondary Improvement (Stanford University) 

• National Center for the Study of Adult Learning and Literacy (Harvard Univer- 
sity) 

• National Research Center on English Learning and Achievement (CELA) 
(University at Albany, SUNY) 

• National Research Center on the Gifted and Talented (University of Connecticut 
at Storrs) 



Since 1990, the centers have been reduced in number, redirected in their missions to 
mirror somewhat more directly the thrust of the national institutes and increased somewhat in 
size. They clearly provided a somewhat different base than the older, more established regional 
laboratories, which was seen by some as an injection of new thinking and a stronger research 
emphasis than the regional labs. 
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Office of Reform Assistance and Dissemination (ORAD). ORAD manages a large 
portfolio of development and demonstration projects ($185 million in FY97 out of a total ORAD 
budget of $245 million). Sixty two percent of those projects have an outcome evaluation 
component. The ten Regional Laboratories were financed in FY97, as shown on Table 4, at a 
$51 million total level to undertake varied and complex missions, which are described later in 
more detail. The National Commission of Professional Teaching Standards was supported at the 
$5 million level in FY97 and the Gifted and Talented Center at $1.5 million, of which $4.4 
million constituted basic and applied research. Other ORAD funds support the work of expert 
panels in identifying promising and exemplary programs, and dissemination activities. 



Table 4 

FISCAL YEAR 1997 FUNDING FOR REGIONAL LABS 



Lab 


States Served 


1997 Funding 


Northeast and Islands Laboratory 
at Brown University 


Connecticut, Maine, Massachusetts, New Hampshire, New 
York, Rhode Island, Vermont, Puerto Rico, and the Virgin 
Islands 


$6.1 million 


Mid-Atlantic Laboratory for Stu- 
dent Success 


Delaware, Maryland, New Jersey, Pennsylvania, and Wash- 
ington, DC 


$5.2 million 


Appalachia Educational Labora- 
tory 


Kentucky, Tennessee, Virginia, and West Virginia 


$4.1 million 


Southeastern Regional Vision for 
Education 


Alabama, Florida, Georgia, Mississippi, North Carolina, and 
South Carolina 


$5.6 million 


North Central Regional Educa- 
tional Laboratory 


Illinois, Indiana, Iowa, Michigan, Minnesota, Ohio, and 
Wisconsin 


$6.6 million 


Southwest Educational Develop- 
ment Laboratory 


Arkansas, Louisiana, New Mexico, Oklahoma, and Texas 


$5.5 million 


Mid-continent Regional Educa- 
tional Laboratory 


Colorado, Kansas, Missouri, Nebraska, North Dakota, South 
Dakota, and Wyoming 


$4.2 million 


WestEd 


Arizona, California, Nevada, and Utah 


$5.5 million 


Northwest Regional Educational 
Laboratory 


Alaska, Idaho, Montana, Oregon, and Washington 


$5.2 million 


Pacific Region Educational 
Laboratory 


American Samoa, Commonwealth of the Northern Mariana 
Islands, Federated States of Micronesia, Guam, Hawaii, Re- 
public of the Marshall Islands, and the Republic of Palau 


$3 million 


Total 




$51 million 



Source: OERJ, 1998. 



The Dissemination Program. The OERI dissemination program of approximately $50 
million in FY97 is dispersed in four different entities. It includes the activities of the National 
Library of Education which encompasses ERIC, an 800 number electronic system and the library 



collection. It includes the outreach and printing activities of the Office of Media and Information 
Services. And it includes a range of dissemination supporting activities of the Institutes and 
ORAD. . 

Foundations 

The foundations, which are estimated to be the second largest source of educational 
RD&D financing after the federal government, sponsor a wide range of education and education- 
related programs and activities, a fraction of which can be considered RD&D activities. Most of 
such sponsorship goes to institutional or student support, and thus is not appropriately treated as 
RD&D. Like the federal government, there is no documented source of foundation-supported 
RD&D. Based on some special studies and anecdotal conversation, we believe that total to be in 
the range of $325-400 million annually. These data collection results are attached, showing first 
annual amounts of commitments or spending of major foundations with education interests 
which can be treated as education R&D, and second major initiatives (several multi-year) of a 
larger group of foundations which relate directly or indirectly to educational research (Tables 
A-2 and A-3). 

Within the foundation funding, one can find some clear examples of sponsored basic 
research, but the bulk of it from its description would fall in the categories of applied research 
and dissemination. Much of it is seeking improved quality of education in some dimension for 
all or for particular populations. It is also the source of financing for educational reform, in some 
cases for substantial demonstration projects. 

3. Who Are the Performers? 

Education RD&D is performed by individual consultants and researchers and by a wide 
range of institutions. These institutions include universities and other educational institutions, 
non-profit and for-profit research organizations, a growing number of intermediary technical 
assistance organizations, and governmental agencies. In this section, we will concentrate on the 
OERI governmental structure and universities. Discussion of the intermediaries will come in a 
later treatment of dissemination. 

OERI 

The OERI structure is portrayed on Chart 1 below. For this discussion, we will concen- 
trate on the National Institutes and the Office of Reform Assistance and Dissemination (ORAD). 
While NCES is appropriately considered part of the knowledge building system, it is not integral 
to the RD&D system, and thus not treated as a performer. The same judgment is made about 
other components of OERI. 

National Research Institutes. Goals 2000: Educate America Act in 1994 established five 
National Research Institutes within OERI: 
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Chart 1 



OERI ORGANIZATION 




• The National Institute on Student Achievement, Curriculum, and Assessment 

• The National Institute on Postsecondary Education, Libraries, and Lifelong 

Learning 

• The National Institute on Governance, Finance, Policy-Making, and Management 

• The National Institute on the Education of At-Risk Students 

• The National Institute on Early Childhood Development and Education 

Vinovskis (1997) noted that the Institutes have pursued their own research agendas rather 
than working on an integrated strategy. He surmised that this may be due to the fact that there is 
no one who oversees an overall research agenda. Each of the Institute directors reports directly 
to the Assistant Secretary. 

The National Research Institutes funded 1 1 research centers in fiscal year 1997. 
Vinovskis (1997) noted that many of the centers fund too many small projects that are not well 
integrated. The centers pursue basic and applied research and dissemination rather than the long- 
term development of education materials and models. 
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Office of Reform Assistance and Dissemination (ORAD). The Office of Reform 
Assistance and Dissemination has four divisions: 

• State and Local Support Division (SLSD) 

• Knowledge Applications Division (KAD) 

• Development and Demonstration Programs Division (DDPD) 

• Learning Technologies Division (LTD) 

In March 1997, ORAD had 61 staff members (Vinovskis, 1997). ORAD includes the 
regional education laboratories and ERIC. ORAD is also responsible for identifying promising 
models, development and testing them, and disseminating the information (Vinovskis, 1997). 

Regional Laboratories. The Regional Laboratories were developed in 1965 as part of the 
Elementary and Secondary Education Act. In 1994 the program was reauthorized to “promote 
knowledge-based school improvement to help all students meet high standards and to help the 
nation meet the National Education Goals.” The goal of these ten labs is to provide information 
on research and knowledge from practice to individuals working to improve education (OERI, 
1996). Two new goals established in the 1994 legislation were bringing together successful state 
and local reform efforts, and scaling-up successful reforms to be adopted (Vinovskis, 1997). 

Each of the ten labs is governed by a board that represents teachers, researchers, and 
policymakers in the region. The functions of these regional laboratories include: 

• providing information, training, and technical assistance, and developing or 
implementing programs, 

• developing models for systemic reform and expanding exemplary reforms, 

• improving access to research and best practices, 

• collaborating with the community in development and dissemination, 

• working with other ED-funded technical assistance providers, and 

• bonding with the research community. 

Below we briefly describe the specialities of each of the ten regional labs. 

Northeast and Islands Laboratory at Brown University (LAB): The three areas of assis- 
tance that LAB focuses on are teaching and learning, the school environment, and partnerships 
for systemic reform. LAB attempts to impact schools directly by developing curricula; providing 
support for students from language minority, urban, and disadvantaged backgrounds; providing 
professional development; working with the community on school improvement; and promoting 
technology in the schools. LAB has a specialty in language and cultural diversity and engages in 
applied research aimed at meeting the needs of language and culturally diverse students. 

Mid- Atlantic Laboratory for Student Success (LSS): The goals of LSS research and 
development are to identify effective educational practices and policies currently being used, 
create new classroom instruction methods, and improve school environments to increase stu- 
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dents’ chances of success. LSS has a special focus on urban education, and attempts to improve 
the success of children in urban schools. The Services to the Field Unit provides access to infor- 
mation services, technical assistance, and training resources; links to existing system of dissemi- 
nation and professional development resources; and collaborative research with practitioners. 

Appalachia Educational Laboratory (AEL): AEL is conducting a research and develop- 
ment project with each of the states it serves. The foci of these various research projects are 
student writing, Internet-based instruction, technical assistance for improving student achieve- 
ment, and using the education information system to inform instructional decisions. AEL’s 
specialty area is rural education. They produce an annual paper on rural education issues, an 
electronic journal, and a rural education digest. 

Southeastern Regional Vision for Education (SERVE): SERVE’s applied research and 
development projects include indicators of school progress, a self-assessment process for high 
schools, teacher evaluation measures, and professional development opportunities for teachers. 
SERVE also organizes study groups to examine promising programs and how to scale them up. 
SERVE is conducting a national assessment of critical needs in early childhood education, its 
specialty area. SERVE-Line provides educators with a networking tool that allows them to share 
resources. SERVE has also created an internet service which provides educators with searches of 
educational literature. 

North Central Regional Educational Laboratory (NCREL): NCREL has five centers 
working on applied research and development. The Teaching and Learning Center collects 
information, researches, and disseminates research on teaching and learning. The Center for 
School and Community Development helps schools plan and make decisions by identifying 
research and models of best practice and by providing technical assistance, training, and 
consulting. The Center for Scaling Up aims to improve knowledge about how to replicate 
proven programs and practices, and helps in the implementation. The Evaluation and Policy 
Information Center develops networks of researchers, tracks reform efforts, conducts short-term 
research to influence education policy, and holds online policy seminars. The Technology for 
Educational Achievement Center, working onNCREL’s specialty area, studies technologies to 
determine their potential benefits in the schools and examines policy issues of applying new 
technology to education. 

Southwest Educational Development Laboratory (SEDL): SEDL’s research, develop- 
ment, and dissemination goals include increasing family and community involvement in 
education, addressing diversity, promoting policy development, improving instructional systems, 
applying technologies to education, and altering the organization of schooling. The speciality 
area of SEDL is language and cultural diversity. In this area they will examine effective school 
characteristics, the teaching and learning environment, development of school staff competen- 
cies, native language development, assessment methods, and parental and community involve- 
ment. 
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Mid-continent Regional Educational Laboratory (McREL): McREL’s four research 
programs aim to improve standards, curriculum, assessment, and instructional strategies; identify 
human development and motivation factors that improve learning; develop tools to improve 
organizations; and increase knowledge about the effect of educational systems on systemic 
reform efforts. Their field services activities include publications, technology networks, data- 
bases, distance education networks, meetings, training, technical assistance, and dissemination. 
McREL’s specialty area is curriculum, learning, and instruction. They are working to synthesize 
standards developed nationwide. 

Uniting the Far West Laboratory for Educational Research and Development and the 
Southwest Regional Laboratory (WESTED): WESTED’s applied research and development 
efforts focus on whole school reform, language and cultural diversity, and early intervention. 
Whole School Reform aims to help schools become coherent organizations by using assessment 
to alter current school culture, and adapting and implementing three Johns Hopkins University 
school improvement models! Talent Development Schools, Success for All, and Roots and 
Wings. Language and Cultural Diversity aims to improve schools’ treatment of diverse students 
through professional development. Early Intervention, Care and Education aims to integrate 
education with other social services, private organizations, community groups, and family 
members. WestEd’s specialty area is assessment and accountability, and they aim to use 
assessment to measure and improve students’ performance. 

Northwest Regional Educational laboratory (NWREL): NWREL focuses on assessment 
and accountability, early childhood education, rural education and school change processes. In 
their specialty area, school change processes, NWREL convenes practitioners, policymakers, 
assistance providers, researchers, and program leaders to learn about processes for changing 
schools. 

Pacific Region Educational Laboratory (PREL): PREL has brought together a team of 
educators from public, private, and higher education to conduct applied research and develop- 
ment. PREL’s specialty area is language and cultural diversity. PREL works with the other two 
laboratories assigned this specialty area to develop a complementary program of research in this 
area. 

From this discussion it is clear that the research laboratories are attempting to achieve a 
great deal considering their limited budgets. Additionally, many of these labs overlap in the 
types of research efforts they are conducting. 

Institutions of Higher Education 

Institutions of higher education are the places where most educational research gets done. 
Institutions fund research on education through internal grants, dissertation support, research 
assistantships, leave time for research, faculty travel funds, computers, other equipment or 
supplies, and seed money to start projects and generate outside support (James, 1992). Accord- 
ing to McCarthy (1990), academia has not been a relied on source for education policy, as 
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policymakers have traditionally relied on information from their staffs, lobbyists, and state 
education staff. The reason for academic’s lack of influence may be due to their perceived lack 
of knowledge and interest in problems schools are facing. Even when education researchers do 
work on issues of interest to policymakers, the results are not often presented in a way that they 
will be adapted by policymakers. 



University-Based Education Policy Research Centers. A particular form of university- 
based education research has been the centers which provide research findings on education 
matters and policy options to policymakers and education leaders. The purpose has been to 
bridge the gap between researchers, policymakers, and practitioners. In 1990, 18 such centers 
existed. The centers have been envisioned as a method by which to join researchers, policy- 
makers, and practitioners in solving the problems facing schools. Most have a primary mission 
to “inform the policy-making process by presenting nonpartisan, research-based information to 
policy-makers ” They may compare the advantages and disadvantages of different policy 
options, and evaluate the effect of various initiatives. Centers also conduct studies; host 
conferences, workshops, seminars, and forums; track reform efforts; provide technical assistance 
to teachers and state departments of education; and monitor data to identify new education issues 
(McCarthy, 1990). 

4. Who Sets the Priorities and How? 

In examining the system for annual expenditures in the range of $1 billion on educational 
R&D, it is reasonable to consider who sets priorities and how it is done on either the macro or 
micro level (e.g., national and within particular institutions of importance). This consideration 
begins with enumerating the potential actors in the process. 

Obviously, the most directly concerned are the sponsors of the research and the research 
performers identified in answer to the previous question. The sponsor described in the answer to 
the second question are primarily the federal government and the foundations, each containing a 
large number of individual institutions with substantial or complete autonomy. Even within 
large individual federal institutions such as ED, DOD or HHS, there are subordinate organiza- 
tions involved with educational RD&D which operate with considerable independence. 
Individual foundations, likewise, have total freedom of action, though more have chosen in 
recent years to collaborate around large initiatives. 

Beyond the directly involved sponsors and performers, there are other directly affected 
groups which have substantial stakes in the educational RD&D activity. These include policy- 
makers at all levels, managers of educational institutions, teachers, educational support vendors, 
and the consumers— students, parents and the public. The policy-makers include those with 
legislative and executive responsibilities at all levels — both public and private. The managers 
include those in both private and public institutions; and the educational vendors of particular 
interest are the textbook publishers and the assessment and testing organizations. 
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Together, this is a potentially enormous group who can contribute to and have a stake in 
the results of education RD&D activities — enough to fill one or more concert halls. Education, 
like other areas, faces extended constituencies, making participatory processes complex, time 
consuming and difficult. Somewhat unlike other areas, education approaches the processes with 
a somewhat different mind set than other functional areas. It is more ready than other areas to 
grant all interested parties equal status with not only advisory responsibilities, but also decision- 
making power on a very wide range of RD&D matters, on at least an implicit assumption of 
equal competence. Other functional areas are likely to be equally inclusive in advisory activities, 
but somewhat more tailored in decision-making to match responsibilities and competence with 
the substance to be decided. 

As a general matter, priority setting in educational RD&D is a highly fragmented process. 
Within the federal process, there is no formal process other than the budget to link the agencies 
together, and then only if OMB staff or an Administration initiative take special steps to do so. 
We are unaware of any recent effort to do so, leaving whatever informal interagency coordina- 
tion to the primary agency actors. Periodic informal efforts are made. 

At the Department level, a more substantial effort has been and is being made through the 
mechanisms of strategic plan and the performance measurement system prescribed by the 
Government Performance and Results Act of 1993 (GPRA). This statute now moving into 
serious implementation calls for each department and agency to submit a five year strategic plan 
to OMB and Congress (first formally submitted by ED in the fall of 1997) followed by perfor- 
mance indicators for assessing progress toward goals and objectives (formally due to Congress in 
January 1998, but already submitted in part by ED). This process provides an additional device 
through which the Department can coordinate and integrate its knowledge-building activities. 

The Department has received and initially adopted a recommendation to include an annual 
analytic agenda as part of its strategic planning process to focus on key knowledge-building 
requirements over the next five year period (Morrill and Weiss, 1997). 

Given the requirements of the Educational Research and Improvement Act of 1994 for 
the Assistant Secretary of OERJ and the National Educational Research Policy and Priorities 
Board (NERPPB), there is apparent interest for the Assistant Secretary and NERPPB to 
coordinate its efforts with those of the Department in meeting the requirements of GPRA with 
respect to knowledge-building activities. There would appear to be substantial mutual benefits in 
NERPPB being able to better extend its reach beyond OERI programs and the Secretary’s ability 
to make better known to the research community the Department’s sense of research needs. 
Collaboration rather than confrontation would appear to route to more positive outcomes. 

In pursuing this direction, NERPPB faces two kinds of problems in the existing RD&D 
system. The first is a deeply in-grained distrust between the Department’s RD&D community 
and perhaps a significant share of the external research community on the one hand and the 
Secretary and his staff and a significant share of the political community on the other. In 
caricature, from the research perspective, the research office and community is engaged in the 
production of truth and light for national consumption, while the Secretary and staff are con- 
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sumed with Departmental programs, Administration initiatives, and politics with little regard for 
the knowledge base. From the other end of the telescope, the scene is different for Departmental 
leadership struggling with a set of difficult substantive and political problems to make incremen- 
tal improvements, while the research arm of the Department continues to drift in irrelevance, 
minutae and unresponsiveness. Efforts have been and are being made to lower that distrust, but 
continued progress will be important to effective processes in priority setting. 

A second problem arises from NERPPB’s mission to go beyond the boundaries not only 
of OERI work and the Department, to include the national educational RD&D agenda and its 
priorities. There are no formal processes for such an undertaking, and should not be (the statute, 
while calling for a national perspective, is modest in its expectations). While some federal 
mission agencies come close to such dominance in their area, (e.g., Defense and space), few 
others dominate the national RD&D sponsorship. And for the intellectual health and independ- 
ent inquiry, such dominance is not a desirable goal, even though it simplifies priority setting. 

Notwithstanding the value of decentralization, there are less formal steps that can be 
taken to help shape priority setting. One such step is a periodic statement of priorities as 
NERPPB now prepares. A second approach is through careful dialogue and collaboration with 
other sponsors. Other agencies seem far more aggressive than ED in dialogue with foundations 
about collaborative agendas. 

5. Where Are the Incentives? 

In the RD&D System Committees workshop in September 1997, several of the speakers 
raised the question of incentives, and pointed to the apparent lack of incentives to innovate and 
make use of the products of research and development. Other questions touching on incentives 
were raised such as the observed small investments in R&D made by textbook publishers relative 
to those made by health care product companies. It thus seems appropriate briefly to summarize 
the apparent incentives for the conduct and use of quality RD&D so that they may be considered 
in the committee’s subsequent work. 

• Incentives for the education system at the operating level. While the incentives 
are considered positive at the abstract rhetorical level, it is less clear that they are, 
in fact, uniformly positive. Change is unsettling; and education is a community 
awash in new ideas and fads. Teachers live very full lives with little time for 
reflection and, sometimes, professional development. And school governance is 
full of layered power centers, in which presumed hierarchial relationships are 
more in name than in fact. Thus, the market for innovation and easy acceptance 
of new research is mixed at best. 

• Incentives for the knowledge-builders and data gatherers. The low levels of 
educational research spending have not historically encouraged researchers to 
enter the field, particularly the absence of substantial empirical studies and basic 
research. The growing concerns with the status of education appears to be 
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attracting more good quality researchers to educational issues. This trend could, 
no doubt, be strengthened by more funding and increasingly higher standards in 
research quality. 

• Incentives for educational materials and service providers. It has been noted that 
textbook publishers and test providers put relatively little resources into research, 
relative to drug companies in the health field. The analogy is interesting, but not 
compelling without consideration of monetary incentives. Drug companies are in 
a position to recapture their very large R&D expenditures through pricing and 
patent protections which allow recovery of such costs along with handsome 
profits for their successful products. It is not so clear that copyright protections 
and the competitive environment permits textbook publishers to follow a compar- 
able course. Further, the recovery of the R&D on drugs from the consumer of the 
drug is a more comfortable notion than the recovery of educational R&D only 
from the purchaser of particular textbooks. 

• Incentives for policy-makers and consumers. In the case of both policy-makers 
and consumers, there is reason to believe that solid innovation and quality 
educational R&D is, on balance, positive. However, the ideological load on 
education is such that change of whatever kind can and will be regarded with 
hostility by some. 

Thus, the incentives for innovation and research in education can be regarded as mixed 
over-all. It is certainly less positive than the health field where the new and innovated is 
regarded as an unalloyed good, much in demand by the public. 

6. Given the Substantial Dissemination Budgets, Why is So Little Implemented 
Large Scale? 

This perplexing question has been addressed in the literature and elsewhere with a wide 
range of answers. Some are inclined to fault the content and quality of underlying research; 
others fault the dissemination for the poor results; and one might add that the culprit could lie in 
the barriers that exist within educational institutions, particularly their governance systems. 

Underlying Research. Those that fault the underlying research have identified a wide 
range of targets: 

Misdirected Content 

Slavin (1997a) states that while Federally funded educational research has produced 
much information that may be useful to educational practice, little of the research has been used 
to create widespread program change and major reform. 
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Low Federal Investment in Education RD&D 

Slavin (1997a) suggests that education research has not had the influence of research in 
medicine or science because of a relatively low Federal investment. 

Federal Policies 

Slavin (1997a) argues that education research has not had a large influence on policy 
because of federal policies which prescribe staying away from curriculum development. 

Lack of Training and Technical Assistance 

Slavin (1997a) states that education research is not implemented often due to a lack of 
training and technical assistance. 

Low Quality of Research 

Kennedy ( 1 997) asserts that education research has not impacted teaching because the 
research is not persuasive or authoritative; the quality of the research is not good enough. Slavin 
(1997b) argues that the lack of confidence in education research is a cause of its lack of influ- 
ence, “We can never have meaningful progress in educational programs until we can have the 
same level of confidence in them that physicians can have in procedures or medications passed 
by the Food and Drug Administration.” 

Research is not Practical 

Kennedy (1997) says that the research often cannot be applied because it is not practical, 
and does not address teachers’ questions or constraints. 

Research is not Presented for Teachers 

Kennedy (1997) says that research often is not adapted because is not expressed in ways 
that teachers can comprehend. 

Education System is Too Rigid or Too Unstable 

Kennedy ( 1 997) states that research has not had large impacts on teaching because the 
education system is rigid, or conversely it is unstable and too willing to follow fads. 

Absence of Field Initiated Research 

Hawley (1990) asserts that an absence of field-initiated research negatively effects the 
spread of research findings throughout the nation. 

Dissemination System. Others find the dissemination system dysfunctional in important 

ways. 



According to Klein (1992), the current dissemination system is a haphazard combination 
of many different previous attempts to improve dissemination. Louis (1992) also argues that the 
dissemination system is composed of uncoordinated, or even competitive activities. 
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Klein and Gwaltney (1991) argue that different dissemination programs are often not 
coordinated with one another. Klein (1990) studied 100 education-related clearinghouses and 
found that other than the 1 6 ERIC ones, they did not coordinate information or services on a 
regular basis. White (1990) also noted the fragmentation of the education dissemination system. 
Some of her examples cited by Klein and Gwaltney (1991) were the lack of connection between 
OERI-funded R&D centers focusing on at-risk children and Program-based dissemination 
activities including the bilingual education clearinghouse, the bilingual state capacity building 
grant Program, the Chapter 1 Technical Assistance Centers, or the Early Childhood Technical 
Assistance System. She also found that dissemination activities of the Regional Labs were not 
linked to other dissemination Programs of the Department. Louis ( 1 992) also noted that the 
Regional Labs have been isolated from other dissemination efforts. White (1990) argues that 
such lack of coordination and communication is a problem because effective techniques are not 
shared and consumers may receive contradictory information from different parts of ED. 

The coordination problem does not exist only within ED, but within other federal 
agencies conducting education research as well (Klein and Gwaltney, 1991). Often competition 
is the rule rather than coordination. Materials from one agency are not regularly disseminated 
with the programs of another, and staff communication is rare. They cite several examples, 
including: 

• The Department of Health and Human Services does research on learning, family 
structure, integrated service delivery, and funds dissemination activities related to 
education. 

• The Department of Labor funds research on dropouts and illiteracy, and funds 
dissemination activities related to education. 

• The National Endowment for the Humanities funds research and dissemination on 
students knowledge of history and the humanities. Only recently has some 
cofunding of activities occurred. 

• The National Science foundation has worked on the teaching of math and science. 
Only recently has some cofunding of activities occurred, but more is planned. 

Christopher Cross, former Assistant Secretary of OERI, did begin a movement toward increased 
coordination between different programs at ED and with other agencies. 

Biddle (1996) also argues that a lack of organization in our dissemination system is a 
reason for the lack of use of education research. He states that even when the research we need 
has been done, “we are unable to find reports of it amidst the confusing oversupply of informa- 
tion in our mass society.” 

Klein and Gwaltney (1991) argue that a nationwide dissemination system should be 
developed. This system would contain information on: 
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• education programs, policies, contracts, grants, and data 

• education literature 

• promising and effective or exemplary education programs, products, practices, 
policies, and public information (selected on quality) 

• syntheses and interpretations of education research and knowledge 

Hawley (1990) argues in favor of an “interagency coordinating council for education- 
related research.” This council would be chaired by the Secretary of Education and would 
maintain a database on all education research currently being funded. As well as increased 
coordination, OERI should allow for long-term research projects and ensure stability in funding 
of education research. 

The concern about the dissemination system has taken an interesting recent focus with the 
emergency of intermediary organizations in connection with major reforms efforts in the K-12 
system, such as New American Schools and the reforms using the work of Robert Slavin. These 
intermediary organizations combine the functions of research and technical assistance in a 
continuous, hands-on interaction with the school system attempting comprehensive reforms. The 
perceived need for and early positive indicators from this process suggest that effective dissemin- 
ation of educational research related to major reform may need a far more proactive form of 
dissemination to achieve substantial implementation. 

Educational operational governance. The preceding observation leads to at least a 
hypothesis that the complexity and dispersal of power within school systems among teachers, 
principals, superintendents, school boards, and other staff is such that the local governance struc- 
tures themselves defeat large scale reform efforts based on R&D. While not written about quite 
in this way, this hypothesis would appear to be at least an implicit assumption in those advocat- 
ing voucher systems extending to private schools. 

7. Who Talks to Whom About What? 

We have discussed elements of the communication problem in response to earlier ques- 
tions about spending, performers and priority setting. We return to the communication question 
due to its salience in the identification of system problems and development of remediating 
strategies. One can think about these additional communications issues at three different levels: 
(1) communication among the knowledge -builders; (2) communication among the knowledge- 
builders, the sponsors and policy-makers; and (3) communication between knowledge-builders 
and the educational operating institutions. 

Among knowledge-builders. As we will note in the following definition section of this 
paper, the several different disciplines which are appropriately brought together in educational 
R&D have different traditions and standards as to what constitutes knowledge in the knowledge 
base. Yet it is important for educational practitioners to know with what certainty, caveats and 
limitations they should take the results of educational R&D. This exceedingly important, but 
equally difficult, topic is not often the subject of discussion among knowledge-builders. Further, 



when raised, it is often dealt with in terms of the peer review process. Peer review is an 
important component of quality assurance; it is a necessary, but not sufficient condition with 
respect to assurance of the validity, replicability and scaled-up potential of micro-experimental 
educational innovations. And it is in this set of educational R&D where the quality of the results 
are particularly important. In terms of the improvement of the RD&D system, this is an area of 
special relevance for the NERPPB. 

Among knowledge-builders, sponsors and policy-makers. The conversations among this 
group often tend to be about the wants and needs of their respective communities in absolute and 
particularistic terms. What problems should be attacked? What ideas should be tried? What 
resources should be committed to specific projects? It is a conversation periodically tinged with 
suspicion or skepticism. Such conversations focus less often on aggregate effort, strategic 
considerations and collaborative undertakings. Could there, for example, be a more collaborative 
effort to identify an integrated set of policy and knowledge-building goals with a set of specific 
objectives, resources and indicators of success? Under selective circumstances, there are 
periodic instances of such strategic planning in the United States about other topics, including 
those beyond national defense. The new GPRA framework provides some potential for that to 
occur, which moves in the direction of the creation of learning communities. 

Between knowledge-builders and educational practitioners . In the definition section 
which follows, we undertake to define effective dissemination that necessarily focuses on the 
conversation between knowledge-builders and educational operators. Other reviewers have 
found much to criticize about that conversation, which we later record. Here, we wish to sum- 
marize a few key points about the dialogue. 

From the practitioner perspective, the recurring central complaints focus on the trans- 
ferability or inapplicability of the research and knowledge to practitioner operating conditions or 
the inaccessibility of the needed information. For those practitioners involved in comprehensive 
reforms, the fragmentation of the knowledge base is an added complaint. From the researcher 
perspective, the unfamiliarity of practitioners with research imperatives and the impenetrability 
of education’s institutional structures complicates communications. This persistent problem 
raises questions about the present character of the dialogue, and ways that it might be changed 
for the better. 

II. Definitional Issues 

Having described some broad characteristics of the education RD&D system, it is time to 
return to some central definitional issues which contribute to the mapping of the enterprise. The 
way one structures these definitions shapes the direction one is likely to take in assessing the 
system and its results. 
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A. Basic Research Versus Applied Research 

“A new drug or medical procedure requires enormous investments in applied research in 
preparation for a rigorous FDA approval process, but builds on basic research. A similar rela- 
tionship between basic and applied research in education, leading to products of great benefit to 
children, could give all of education R&D the status and funding it deserves” (Slavin, 1997a). 

Comparison among fields and disciplines about the distinctions between basic and 
applied research can sometimes lead to difficulties, and usually to arguments. Basic physics, the 
health sciences and medical practice, and education all start from somewhat different places in 
assessing what is basic versus what is applied research. Nonetheless, basic research is directed to 
a fundamental understanding of the phenomena (what is matter? what is cancer? how do people 
learn?) in a structured and replicable way; while applied research is directed to application of that 
knowledge to particular purposes (can we develop a new source of energy? is there a way to 
prevent cancer? how should teachers manage a class?). 

Until the work began in earnest on cognitive science, most education research has been 
directed to the applied side. The National Research Council report of 1991 and other assess- 
ments point to the contributions now possible and being made by advances in basic cognitive 
science. It still remains the case that most education research is devoted to applied work, yet far 
from all desirable knowledge has been acquired, particularly in the area of applying basic 
research to classroom settings. 

B. Research Versus Evaluation Versus Data Collection 

In dealing with applied social science research, there is a tendency to treat research, 
evaluation and data collection as quite different things. Yet all three share much in common with 
respect to purpose, value and even techniques. All seek to provide valid information to guide 
action and future knowledge-building activities. While research may explore new ideas for 
future policies, evaluation assess current programs, and data collection document events and 
progress, all can broaden and deepen the knowledge-base about a given topic. Further, they 
often share methodologies in the production of quality results. 

Thus, the distinctions among them may be important for some purposes, their common 
threads suggest that they be thought about in a comprehensive way when developing knowledge 
building strategies. (Morrill and Weiss, 1997). 

C. Applied Research Versus Policy Research 

Another somewhat vexing set of definitional distinctions center on applied versus policy 
research. The distinctions between them can be real, but the usual basis for making the distinc- 
tion has more to do with issues of turf, nuance and politics. 
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Applied research and policy research may be genuinely different in that the applied 
educational work may affect the knowledge base or the operational practice of teachers without 
having an explicit or immediate connection with broader educational policy at any level, while 
all “policy research” is presumably undertaken at least in part to draw out the policy implications 
of the general knowledge base or some specific piece of applied research. This distinction, while 
real, is not usually what the conversation is about when the distinction is made. More often, the 
issues at stake in making the distinction have to do with whether the policy community is paying 
the slightest attention to the knowledge base in devising new policies, or whether the policy com- 
munity is twisting the knowledge base to misrepresent its policy implications, or both. 

The education community can be seen from other fields and disciplines as near obsessive 
about “politization” of education and educational research from both ends of the two perspectives 
described above. From an OERI perspective, the issue can be caricatured as “pure, but ignored” 
or “involved, but political.” The interaction between research and policy is bound to produce 
some tensions, but in other venues, sorting evidence from conviction can be dealt with in sound 
ways and less emotional environments (Morrill and Weiss, 1997). This is a matter to which the 
project will return in later papers. 

D. Knowledge-Building - The Standards of Evidence 

According to Shavelson (1988), the “truth test” or whether the research is trustworthy, 
will play an important role when attempting to change the ways that policymakers concep- 
tionalize an issue. The “utility test” or whether the research provides direction to the agency for 
current practice or alternative approaches, should also be applied. 

Sroufe (1997) argues that there are not enough studies done which are of high enough 
quality to influence national education policy. He cites a 1997 GAO report which examined 
hundreds of studies and found that only 22 were satisfactory. Many of the others did not have a 
suitable comparison group, and none used a nationally representative sample. 

The difficulty of the definitional issues described above also hinges on the standards of 
evidence implicitly or explicitly being used to assess the quality of knowledge, the confidence 
with which one can act upon research findings, or the minimum standards upon which any action 
will seem warranted. The social science community tends to take refuge in the peer review 
processes employed in other sciences as a guarantor of research quality. While such processes 
have substantial strength and few easy alternatives in most applications, the standards of 
evidence between disciplines in the social sciences are quite different, much less the differences 
which no doubt exist among the policy, practitioner and other lay communities and the research 
community. 

This is not to suggest that there is only a single standard which should be uniformly 
applied, but rather to suggest that there needs to be: (1) clarity as to what standards are being 
applied; and (2) where along a spectrum from unambiguous outcomes from random selection 
experimentation through compelling correlations to “clinical” conviction, the standard of 
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evidence is to be set. This selection should be made in the context of certainty that the ultimate 
standards are unavailable in all circumstances for reasons of cost, time, and moral imperatives. 

E. Dissemination - What Constitutes Effectiveness? 

Above, we discussed reasons for the perceived failure of the educational dissemination 
system. Discussions of the failure of the system rely on ideas about what the ultimate goals of a 
dissemination should be, and what are reasonable expectations of such a dissemination system. 
Here, we address the issues of what a dissemination system should look like, what may be 
reasonable expectations for such a system, and what should be the ultimate goals. 

According to Klein (1992), there are five important components of a good national 
education dissemination system, some of which do not exist. These are: 

• Education resources : strategies to identify and disseminate the highest quality 

resources in each of the following categories do not exist. There are many 

interaction effects between the knowledge and the vehicle chosen (Klein and 

Gwaltney, 1991). 

► products such as texbooks and videos, instructional or management 
programs, 

► educational practices or principles used in classrooms, 

► public policies which are focused toward the accomplished of particular 
organizational goals, and 

► public information activities such as the sharing of research and statistics. 

• dissemination functions : 

► spread, the one-way distribution of information aimed at increasing 
awareness. This is the most common type of dissemination used by the 
Department of Education, and is usually performed by different offices 
and programs. While the Departement of Education does review these 
activities, there is no system to identify which materials should be spread 
to which users, which functions may fill a need, and which may be most 
cost effective. 

► choice, the dissemination of information on different options, often carried 
out by libraries, clearinghouses, referral or information centers, data bases, 
and catalogs. ERIC does not include computer software, audiovisual 
materials, expert referrals, and most books. Nor does ERIC aid users in 
identifying effective resources or choosing between various resources. 

► exchange, the flow of information in many directions through the use of 
such tools as forums, site visits or meetings. Additionally, there are ED 
sponsored teleconferences and computer networks. There has not been 
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much emphasis on aiding users sharing exemplary resources or evaluating 
their own resources. 

► implementation, the use of knowledge which includes technical assistance 
or training. This is the focus of OERJ’s regional laboratories. This should 
not only include the replication of effective practices, but also the 
improvement of capacity and effectiveness through training schools to use 
research and solve problems (Klein and Gwaltney, 1991). Hawley (1990) 
asserts that implementing agencies usually must be provided with techni- 
cal assistance and training models to ensure adoption and implementation 
of even the most promising researched ideas. 

• Bureaucratic governance levels : coordination among these different levels is very 
limited. 

► federal 

► regional-multistate 

► state 

► intermediate 

► local district 

► school 

► community 

• Content and population focus : Louis (1992) argues that this structure has led to 
competition between the two different types of disseminating units. The system is 
complex and disorganized. Therefore, users have experienced great confusion in 
where to go for assistance. 

► multifocus, such as the Educational Resources Information Center system 
(ERIC) and the Regional Educational Labs. 

► categorical programs, such as the dissemination activities which are part of 
the special, vocation, and bilingual programs. Many of these programs 
have identical dissemination functions, but they do not coordinate their 
dissemination or leam from each others’ experiences. 

• Dissemination roles and skills: 



► librarians 

► technical assistance providers or trainers 

► publishers 

► media producers 

What is the ultimate goal of the educational dissemination system? We would argue that 
efforts aimed at supporting school improvement through the dissemination of new information 
(including the provision of technical assistance in its use) must be judged by their ability to 
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demonstrate the effective use of that information in real school settings. Justification for Federal 
support of dissemination efforts, if they do not lead to changes in services to populations of interest 
to Federal education programs, is weak at best. 

Table 5 

KLEIN’S AND GWALTNEY’S EXAMPLES OF DISSEMINATION FUNCTIONS 
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As described above, Federal programs in education use various dissemination approaches to 
provide information about new research knowledge, about new approaches derived from effective 
practices, or about new (Federal) program initiatives, requirements, etc. Although Federal support 
for research should not be conditioned solely upon demonstrations of effectiveness (the null finding 
is frequently knowledge gained), and Federal approaches to identifying effective practices will not 
always identify initially only those that are effective, Federal dissemination efforts should be 
supported only if they disseminate information demonstrated to have resulted in effective practice - 
effectiveness being related to demonstration of improved results for children or more cost-effective 
approaches to achieving similar results - and the dissemination approaches have been demonstrated 
to have resulted in effective knowledge use. 

While restrictive at first glance, this criterion can be thought of in comparative ways. Would 
we tolerate the Department of Health and Human Services’ use of its dissemination funds to support 
the widespread distribution of information about untested surgical practices, or dietary supplements, 
or any of a host of health practices that would be analogous to disseminating information about 
unproved education practices? Or would it be appropriate for the Department of Transportation to 
disseminate information about new untested theories of how to land commercial aircraft, or of 
appropriate air traffic control practices? 

Further, would we accept these agencies using approaches to the demonstration of their proven 
information that lets it sit on library shelves, or that doesn’t tell the user how to apply it in their 
clinical practices? In fact, these agencies use the most sophisticated social marketing techniques 
developed to change consumer behavior (how many of us have changed our dietary, smoking, or 
exercise patterns over the past two decades based on these campaigns) or to change clinical practice. 

While these analogies to other federally funded public activities are instructive, it must be 
acknowledged that education research is fundamentally different, and perhaps more difficult than 
these other fields. Effectiveness of an educational practice is very difficult to demonstrate. In order 
to test the effectiveness of a new teaching methodology or classroom setting, a controlled experi- 
ment must be devised. Students must be randomly assigned to a classroom with the new system or 
without, samples must be large enough, and students should not be aware that they are being placed 
in a different setting as part of such an experiment. Such conditions are usually very difficult to 
fulfill, as while we are conducting such an experiment, we are also affecting students’ lives. If it 
appears, mid-experiment, that a student would be better served in another setting, it is problematic to 
continue with the initial randomized design. Therefore, such conditions are rarely realized. 

Even when such controlled experiments are adhered to and practices are shown to be effec- 
tive, implementation is complicated. As alluded to in the previous point, the educational setting is 
very important. While a particular program may prove effective with certain types of students, 
teachers, and schools, it may not be effective in a different setting or scale. When finding that a 
program or practice is effective, the limitations as well as the potential need to be explicit and 
understood. 

The status of the field of education research means that it needs to be seen in develop- 
mental terms. While the educational dissemination system should concentrate on the practices 
and programs that have been proven effective by the most rigorous standards, it should continue 
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to identify promising potential programs which are proven to work in specific circumstances, and 
develop trials for more extensive and conclusive testing. 

III. Planning and Execution of Education RD&D 

In this and subsequent sections, we undertake to restructure the information and opinion 
provided in the first section around a more conventional structure of analysis starting with an 
overall view of the system. In this section, we first present some broad views held by others, and 
then provide a summarization of the points in the first section concerning the planning and 
execution of federal education RD&D. In succeeding sections, we continue with such summa- 
ries about related topics. 

A. Views of Others 

Some analysts find other functional models such as the agricultural extension service to 
hold promise for education. According to Rogers (1992), the important characteristics of such a 
system would include: 

• Control and participation in the system by the teachers, 

• A focus of research on user problems and needs, 

• The ability to move information both from the researchers to the practitioners and 
from the practitioners to the researchers. Louis (1992) also argues that 
communication must go in both directions. 

• Flexibility to respond to changes in needs and environments, 

• Sensitivity to the organization structure of the U.S. education system. According 
to Louis (1992), “the idea that school development and knowledge use are 
intertwined has not been well incorporated into educational dissemination policy.” 

• Evaluation of educational innovations (because effects will not be as transparent 
as in agriculture). 

• Individuals who would be responsible for translating the knowledge into a form 
appropriate for the users, and 

• Sufficient funding for at least 20 years. Klein (1992b) also argues that a 20-year 
federal commitment would aid in the development of a national dissemination 
system by avoiding political battles. James (1992) also finds problems with 
education research’s small scale, short term, disjointed studies. 



Other analysts have focused most heavily on the dissemination system as a key area of 
needed improvement. According to Klein (1992a), there are three goals that a new education 
dissemination system should work towards. First, the system should aid users in understanding 
the value and importance of different education tools so that the practitioners will use those 
resources which are most likely to improve education. “A nationwide education dissemination 
system will strive to make up-to-date information about high quality resources readily accessible 
and therefore directly serve educators’ information needs.” The current system does not 
disseminate information on the large majority of educational knowledge, and does not aid the 
users in determining which programs, products, practices, and policies will be most useful for 
their particular needs. The second goal is to involve researchers, developers, practitioners, and 
users in all dissemination functions. The third goal is the development of a “dissemination- 
driven RD&D system.” 

Still other analysts concentrate on resources, distributions and process. Hawley (1990) 
states that OERI should spend about 30 percent of its budget on basic research and 20 percent on 
field-initiated proposals. OERI should improve the peer review process and ensure that all of the 
reviewers have sufficient technical knowhow. There should be legislation that requires OERI to 
utilize peer review in the award of most OERI funds. Missions of research centers should be 
defined broadly and contract awards should be made based on the proposed research’s quality, 
not its adherence to prescribed research questions. OERI staff should review center work. 
OERI-funded laboratories should not be prohibited from conducting research. OERI should 
make the amount of funds awarded based on quality of research proposals. Centers should not 
be limited to 5-year awards, but should allow research to extend over longer periods of time, 
during which other contractors could challenge the incumbent center. OERI and the Secretary 
should be clearer about national priorities for education research and discussion on these 
priorities should be cultivated. 

B. Summary of Issues 

A review of planning and execution of the education RD&D system would suggest that: 

• The planning of the educational RD&D is fragmented and partial at most, in both 
a formal and informal sense. While a formal comprehensive structure may be 
neither feasible or desirable, an informal collaborative structure has real potential 
for improved outcomes, particularly if creatively coupled with GPRA require- 
ments. 

• The aggregate spending on educational RD&D is probably substantially less than 
optimum by comparison with any other major activity in the society, perhaps by a 
factor of two or more. Recognition of this deficiency alone will, however, be un- 
likely to result in more resources absent good research ideas, some reform of the 
RD&D system, and more confidence that the results will be better than in the past. 
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• There are sufficient reasons to at least question, if not change, the distributions of 
educational R&D resources along several dimensions. Basic research appears to 
be underfunded relative to applied. The content focus of the research appears too 
heavy on peripheral issues and too light on critical matters of curriculum and 
learning. And there appears to be too much emphasis on research about micro 
innovation and too little on validity and scale-up of promising practice. 

• There also appears to be adequate grounds for concern about the distribution 
between R&D on the one hand and dissemination on the other. These concerns 
are two fold. First, the balance appears to be tilted too heavily toward dissemina- 
tion (though numbers are hard to come by). Second, there is substantial reason to 
worry about the quality and effectiveness of the dissemination system. At least 
for the dissemination of information about comprehensive reform, an essentially 
reactive system appears inadequate to the task. The appearance of intermediary 
organizations raises the possibility of a new, more effective model. 

• There are good reasons to worry about the incentive structure to support quality 
educational RD&D, but no obvious quick fixes. 

• The availability of comprehensive information about the resources in the national 
education RD&D system is appallingly weak. 

IV. Methodological Issues 

The major methodological issues that have been identified in our review of the existing 
system are two: standards of evidence and characteristics of the applied research program. They 
are interrelated. 

As pointed out in the definitional issue section above, there are no commonly understood 
and accepted standards of evidence for assessing the validity of educational innovations and 
improvements. A number of research articles and the September 1997 committee workshop call 
attention to this issue. Some, such as Robert Slavin, would push for sharp change to the point of 
creating an entity like the Federal Drug Administration which would rigorously test and validate 
innovative educational models before they could be advertised as successful or effective. 

Whether one takes quality assurance to that level of regulation or not, the absence of any 
clear standards and identification of research validity leaves practitioners prey to faddism and 
disappointment. Until there is some approach to disciplining the research community and 
informing the practitioners, it is unlikely that the full potential value of competent research 
results will be realized, including the desired sophistication of the sponsor and policy-making 
communities to recognize and demand quality research. It should be clear that this is not solely a 
demand for comparison group demonstration and evaluation, though more of that is surely 
needed. But it is a demand for more realism and disclosure about the limitations as well as the 
promise of micro-experimentation. 
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The second and related issue is the distribution of applied research among theoretical, 
micro-experimentation and large scale trials. As noted in the September 1997 workshop, the 
sponsorship of large scale trials has shifted away from the federal government as sponsor. While 
sponsorship is a separable matter, the quantity of large scale trials with accompanying rigorous 
evaluation appears to be shrinking, while the need may be rising. Further, the absence of federal 
sponsorship of the research and evaluation concerning such trials puts a heavy load on founda- 
tion funding. 

V. Governance Issues 

The structure of the oversight, direction and conduct of education RD&D, as described 
earlier, is exquisitely complex, and, with respect to the U.S. Department of Education, minutely 
prescribed in statute. The institutional scene is (not surprisingly) a reflection of the accumulation 
of many years of battles among the forces and interested stakeholders in the educational field. 

All of these conditions make adjustments in the status quo daunting undertakings. Nonetheless, 
there are reasons to raise questions about the effectiveness or utility of the present governance 
structures, capacities and processes. In particular, there appear to be a variety of mismatches 
between goals and expectations on the one hand and jurisdictions and capacities on the other. 

• The goals or expectations for a national focus in the federal structure (Department 
of Education, OERI and/or NERPPB) are not matched well with the resources or 
authority to exercise that kind of national leadership in any formal sense. Further, 
though research and information have been recognized for many years as appro- 
priate federal functions, the political controversies in education serve to limit the 
scope of action in these long-standing functions. For example, the equally long- 
standing concern about federal involvement in curriculum may well be part of the 
reason for under-investment in this critical educational research priority. 

• The mission statements for the national institutes and the regional laboratories are 
broad and expansive, while their staffing and financial resources are limited, and 
in some cases, declining. Within the federal staffing parameters, there appears to 
be very little way to produce “critical mass”; and external funding budgets are 
comparably small and inadequate for the purpose of producing national leadership 
on issues. At most, the institutes and labs have modest capacity to conduct their 
own work, and therefore must be primarily involved with sponsoring, assisting or 
disseminating the work of others. 

• If more major trials were to be desired on the part of the federal government, the 
present OERI budgets — and potentially the staffing capacity — would not be able 
to mount the effort. The budgets for such work would likely come, in any event, 
from programmatic funds. These circumstances require a closer collaboration 
between OERI/NERPPB, and the rest of the Department, if the former are to play 
an important role in research strategy. 




32 



61 



• While the resources and coordination efforts may prevent duplication of effort, the 
expectations, goals and mission statements between and among the institutes and 
regional labs do overlap. 

• The broadness and overlap in the mission statements suggest a reluctance to estab- 
lish true priorities rather than a “something for everyone” philosophy, which can 
be substantially debilitating when budgets are limited. 

• The institutional separation between the technical assistance organizations and the 
dissemination structure complicates the effectiveness of dissemination, particu- 
larly if more hands-on assistance is required to produce comprehensive reform. 

VI. Integration of the Map 

The foregoing descriptive analysis of the existing education RD&D system provides a 
large array of issues across the full spectrum of the system from its planning, resources, structure, 
processes and selection of priorities to its long history of tinkering with organizational structure, 
complexity and resistance to fundamental change in direction. The map suggests important 
connections among the component issues and a rapidly escalating set of difficulties and 
controversy as the issue agenda expands. Given these trade-offs, there would appear to be three 
different approaches available for the committee’s ultimate report. 

• Comprehensive and explicit. Under this approach, the full range of issues herein 
identified and perhaps others would be directly treated with explicit recommenda- 
tions made about each. 

• Comprehensive and directional. Under this approach, most if not all of the issues 
herein would be identified (and perhaps others), but rather than explicit remedial 
recommendations in every case, the direction of remediation would be indicated, 
with selective recommendations when the committee sees fit to do so. 

• Selective and explicit. Under this approach, several areas of concern would be 
selected for detailed treatment with explicit remedial recommendations made 
about the selected topics. 

While all three approaches are plausible, we are, subject to discussion and determination 
by the sponsors, attracted to the middle course. 
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Table A-l 



DEPARTMENT OF EDUCATION RESEARCH FUNDING 



General Authority Funding 


Name 


Description 


FY 1995 
Funding 


FY 1996 
Funding 


Educational 
Research, Develop- 
ment, Dissemination, 
and Improvement Act 


Authorizes the research, development, demonstration, evaluation, 
dissemination, and technical assistance activities of the Office of 
Educational Research and Improvement aimed at expanding 
fundamental knowledge of education and promoting the use of 
research and development findings in the design of efforts to 
improve education. Included are authorities for five national 
research institutes and the national education dissemination 
system. 


S86.2M 


$97.6M 


Rehabilitation Act, 
Title II, Research and 
Training 


Authorizes the National Institution on Disability and Rehabilita- 
tion Research (NIDRR) to support research, development, dem- 
onstration, and dissemination activities that seek to improve the 
lives of persons of all ages with disabilities, including those with 
the most significant disabilities. 


$70. OM 


$70. OM 


Elementary and Sec- 
ondary Education 
Act, Title X, Part A, 
Fund for the 
Improvement of Edu- 
cation 


Authorizes support for nationally significant programs and pro- 
jects to improve the quality of elementary and secondary educa- 
tion. These include development, demonstration, evaluation, and 
dissemination activities. 


S36.7M 


$36.7M 


Higher Education 
Act, Title X, 
Postsecondary Im- 
provement Programs 


Authorizes the Fund for the Improvement of Postsecondary 
Education (FIPSE) to support demonstrations of exemplary, 
locally developed projects that encourage innovative reform and 
improvement of postsecondary education. 


S17.5M 


$17.5M 


General Education 
Provisions Act, Part 
D, Section 422 


Requires the Secretary to collect data and information to obtain 
objective measures of the effectiveness of programs. Program 
administration funds may be used pursuant to this authority. 


S3.4M 


$3.2M 


TOTAL 




$213. 8M 


$225M 



Source: U.S. Department of Education, 1995. 
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Authorities that Support Major Department Programs 


Name 


Description 


FY 1995 


FY 1996 






Funding 


Funding 


Goals 2000: Educate 


Authorizes under sections 219, 220, 221, 313, and 314 research, 


$0 


S46.5M 


America Act, Title III 


development, demonstration, dissemination, evaluation, and 
technical assistance activities regarding challenging standards 
and systemic education improvement. 






Elementary and Sec- 


Includes authority under part E for evaluations of Title I pro- 


$3.7M 


$1 1.0M 


ondary Education 


grams and demonstration of projects that show the most promise 






Act, Title I, Helping 


of enabling children served under Title I to meet challenging 


$0 


$25. 1M 


Disadvantaged Chil- 
dren Meet High Stan- 
dards 


State content and student performance standards. 






Elementary and Sec- 


Includes broad authority under part A for Federal activities 


S21.4M 


$32. 0M 


ondary Education 


including demonstrations, dissemination, and evaluations of 






Act, Title II, Dwight 


activities carried out under both the Federal and State Eisenhower 






D. Eisenhower 


programs as well as authority under part C for a specific profes- 






Professional 
Development Pro- 
gram 


sional development demonstration project. 


$0 


$3.0M 


Elementary and Sec- 


Includes authority under subpart 2 of part A for rigorously evalu- 


$25. 0M 


$35.0M 


ondary Education 


ated demonstrations of innovative approaches to drug and vio- 






Act, Title IV, Safe 


lence prevention, development and dissemination of model cur- 






and Drug-Free 


ricula and other drug and violence prevention information and 






Schools and Commu- 


materials, and evaluation of drug and violence prevention pro- 






nities 


grams. 






Elementary and Sec- 


Includes authority under subpart 2 of part A for research and 


S7.1M 


$7.3M 


ondary Education 


evaluation, for projects to disseminate information on successful 






Act, Title VII, Bilin- 


bilingual models, and for a national clearinghouse for bilingual 






gual Education, Lan- 
guage Enhancement, 
and Language Acqui- 
sition Programs 


education. 






Elementary and Sec- 


Includes authority under subpart 4 for research related to effec- 


S2.0M 


$7.5M 


ondary Education 


tive approaches for the education of Indian children and adults 






Act, Title IX, Part A, 


and for evaluation of federally assisted education programs from 






Indian Education 


which Indian children and adults may benefit. 






Elementary and Sec- 


Includes authority under part G for the Secretary to reserve funds 






ondary Education 


from amounts appropriated for various ESEA programs (other 






Act, Title XIV, Gen- 


than those in Title I) to carry out comprehensive evaluations of 






eral Provisions 


categorical programs and demonstration projects and studies of 
program effectiveness. 
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Authorities that Support Major Department Programs 


Name 


Description 


FY 1995 
Funding 


FY 1996 
Funding 


Individuals with Dis- 
abilities Education 
Act 


The authorizations for many programs under this Act will be 
expiring at the end of 1995, and new legislation has been pro- 
posed that would extend the current discretionary programs 
through 1996. Many of the current, discretionary, special pur- 
pose programs under parts C, D, E, F, and G include authority for 
research, demonstration, dissemination, and evaluation activities, 
as well as training and technical assistance activities. 


$100. 0M 
est. 


$100. 0M 
est. 


Carl D. Perkins Vo- 
cational and Applied 
Technology Educa- 
tion Act 


Includes authority under title IV for various research, develop- 
ment, demonstration, and dissemination programs. For FY 1996 
the President requested a total of $37.0 for a consolidated na- 
tional program authority. 


S6.9M 


$37.0M 


Adult Education Act 


Includes authority for a variety of research, development, and 
evaluation activities. For FY 1996 the President requested a total 
of $1 1.0M for a new adult education and family literacy national 
program authority, which would include funds for research, 
evaluation, technical assistance, and continued support of the 
National Institute for Literacy. 


S7.6M 


$1 1.0M 


School-to-Work Op- 
portunities Act 


Includes authority under Title IV for the Secretaries of Education 
and Labor to conduct a number of national activities that support 
the work being carried out by States and localities. These include 
the mandated collection and dissemination of specific informa- 
tion, as well as a national evaluation, research, and demonstration 
activities. 


S6.9M 


$15.0M 


Higher Education 
Act, Title IV, Part A 


Includes authority under section 402H for evaluations of Federal 
TRIO programs, which provide postsecondary education out- 
reach and support services designed to encourage qualified 
individuals from disadvantaged backgrounds to enter and com- 
plete college or to be prepared for doctoral studies. 


S1.5M 


$1.5M 


Rehabilitation Act 


Includes authority under section 14 for evaluation of programs 
authorized under the Act. 


S1.6M 


$1.6M 


TOTAL 




183.7 


180.9 



Source: U.S. Department of Education, 1995. 
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Other Categorical Authorities 


Name 


Description 


FY 1995 
Funding 


FY 1996 
Funding 


Elementary and Sec- 
ondary Education 
Act, Title III, Tech- 
nology for Education 


Includes authority under section 3122 for Federal leadership in 
the use of technology in education, including research, develop- 
ment, demonstration, evaluation, and dissemination activities and 
authority under section 3136 for challenge grants to support 
development and demonstration of new applications of technol- 
ogy to education; authority under part B for star schools demon- 
strations of distance education programs; authority under part C 
for development and distribution of educational and instructional 
video programming for preschool and elementary school children 
and their parents; and authority under part D for a telecommuni- 
cations demonstration project for mathematics. 






Elementary and Sec- 
ondary Education 
Act, Title V, Promot- 
ing Equity 


Includes authority under part B for developing innovative strate- 
gies and model training programs in gender equity for teachers 
and other school personnel and the development of policies and 
programs to address and prevent sexual harassment. Also in- 
cludes authority under part C for dropout prevention demon- 
strations and for a national evaluation of the demonstration 
programs. 


S1.7M 

S12.0M 


S1.7M 

$0 


Elementary and Sec- 
ondary Education 
Act, Title X, Part B, 
Gifted and Talented 
Education 


Authorizes support for model projects designed to help educators 
identify and meet the special educational needs of gifted and 
talented students and, where appropriate, to adapt strategies suc- 
cessful with those students to improve instruction for all students, 
as well as support for a national center to conduct research, 
dissemination, and evaluation activities. 


S4.9M 


S9.5M 


Elementary and Sec- 
ondary Education 
Act, Title X, Part C, 
Public Charter 
Schools 


Authorizes support of charter schools, an evaluation of the im- 
pact of charter schools on student achievement, the development 
and dissemination of model State charter school laws and model 
contracts, and the dissemination of information on successful 
charter schools. 


$6.0M 


S20.0M 


Elementary and Sec- 
ondary Education 
Act, Title XIII, Part 
B, National Diffusion 
Network 


Authorizes activities to identify and disseminate effective and 
promising programs and practices. 


$1 1.8M 


S14.5M 


Higher Education 
Act, Title VI, Interna- 
tional Education Pro- 
grams 


Includes authority for research and development of instructional 
programs in international studies and foreign language, including 
development of more effective teaching methods, standardized 
measures of competency, and specialized curriculum materials. 


S23.3M 


S23.8M 


Education of the Deaf 
Act 


Includes authority for Gallaudet University and the National 
Technical Institute for the Deaf (NTID) to carry out research 
activities. 


S5.8M 


S6.0M 
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Other Categorical Authorities 


Name 


Description 


FY 1995 
Funding 


FY 1996 
Funding 


Act to Promote the 
Education of the 
Blind 


Includes authority for the American Printing House for the Blind 
(APH) to develop and distribute educational materials adapted 
for use by blind students in educational programs below the 
college level. 


S0.5M 


S0.5M 


20 U.S.C., 123, 
Howard University 


Authorizes research activities at Howard University, including 
post-doctoral fellowships, primarily in scientific disciplines, the 
purchase of laboratory equipment, laboratory renovation, and 
multidisciplinary research projects. 


S4.6M 


S4.6M 


TOTAL 




S58.6M 


S80.6M 



Source: U.S. Department of Education, 1995. 



Table A-2 

FOUNDATION FUNDING OF EDUCATION RESEARCH AND DEVELOPMENT 



Foundation 


Amount Paid (1996) 
(million $) 


Annenberg Foundation 


20.2 


Carnegie Corporation 


12.7 


DeWitt Wallace 


2.1 


Ford Foundation 


20.3 


IBM International Foundation 


11.0 


MacArthur Foundation 


3.2 


Andrew Mellon Foundation 


16.2 


James McDonnell Foundation 


5.2 


Pew Charitable Trust 


12.8 


Spencer Foundation 


15.5 


Rockefeller Foundation 


11.0 


TOTAL 


- 130.2 
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Table A-3 



INITIATIVES OF FOUNDATIONS TO IMPROVE EDUCATION 



Initiatives of Foundations to Improve Education 1 


Foundation 


Designated Program Area 


Initiatives/Focus 


Funding 

Level 2 






Annenberg National Institute for School Reform at 
Brown University 


$50,000, 000 3 






New American Schools Development Corporation 


$50,000,000 3 


Walter 

Annenberg 


School Improvement 


Education Commission of the States: disseminate work 
of NASDC’s design teams 


$15,000,000 3 






Annenberg Institute: electronically link public schools 


$5,000,000 3 




Restructuring 


New American Schools Development Corporation 


si.ooo.ooo 4 


AT&T 


Teacher Education 


Teachers for Tomorrow: Collaboration to redesign 
teacher education in five cities 


$787, 335 3 


DallCnntU 


Supporting Educators to 
Meet Changing Student 
Needs 


New approaches to teacher education; creative teaching 
opportunities; support for leadership development; and 
support for comprehensive services for students and 
families 


$1,704,500 


DClloOUtn 


Linking Educational Policy 
with Changing Regional 
Needs 


Expand scope of public education; enhance public 
support for education, strengthen education governing 
bodies; and improve accountability and assessment of 
students, teachers, and systems 


$852,000 




Encouraging Educational Ad- 
vanced Through Information 
Technology 


Promote information technology studies and expand 
information technology 


$820,000 






Improve quality of early childhood education 


$3,212,972 


Carnegie 

Corporation 


Education and Healthy De- 
velopment of Children and 
Youth 


Improve educational achievement and health of at-risk 
adolescents 

- Middle Grade School State Policy Initiatives 


$8,455,506 
$2,03 1,000 s 


Education reform including support for the National 
Board of Professional Teaching Standards 


$4,424,000 






Science education 

- American Association for the Advancement of Sci- 
ence: Reformulate content of math, science, and 
technology education 


$2,213,000 
$750, 000 s 
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Initiatives of Foundations to Improve Education 1 


Foundation 


Designated Program Area 


Initiatives/Focus 


Funding 

Level 2 


Annie Casey 


Increasing Public Awareness 


Information dissemination 


$4,286,929 


Program Demonstrations and 
Policy Research 


Comprehensive systemic reform in juvenile justice 
system, adolescent health, and education 


$2,457,030 


Long-Term Comprehensive 
Reform 


New Futures Initiative: Collaborative partnership to 
improve planning and services to at-risk youth and their 
families: Bridgeport, Dayton, Little Rock, Pittsburgh 
and Savannah 


$8,981,500 


Chicago 

Community 

Trust 


Children, Youth, and Fami- 
lies Initiative 


Coordinate and expand families’ access to social and 
recreational services, 10-year project 


$30,000,000" 


Edna 

McConnell 

Clark 


Program for Disadvantaged 
Youth 


Urban middle school reform to improve student 
achievement. Five districts and 12 schools participate 
in the program 


$5,296,041 6 


Common- 
wealth Fund 


Developing Capacities of 
Young People 

- School-to-Work Transition 
Programs 


Manpower Development Research Corporation: Assess 
knowledge about existing school-to-work transition 
programs in 15 communities 


$400,000" 


Danforth 


A Good Beginning for 
Every Child 


Early childhood education, development of parenting 
skills, and professional staff development of early 
childhood and primary school personnel 


$446,942 


School, Family, and Commu- 
nity Partnerships 


Collaboration to assure comprehensive services to 
students and families in need 


$1,019,942 


Leadership for Schools 


Professional development programs 


$2,525,428 


Exxon 

Education 


Mathematics Education Pro- 
gram 


Improve mathematics instruction in grades K-3 through 
the use of math specialists 


$9,200,000 


Elementary and Secondary 
School Improvement 
Program 


School restructuring and teacher preparation 
- New American Schools Development Corporation 


$3,000,000 
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Initiatives of Foundations to Improve Education 1 


Foundation 


Designated Program Area 


Initiatives/Focus 


Funding 

Level 2 


Ford 


Urban Poverty: Children, 
Youth, and Families 


Early childhood, employment, and training opportuni- 
ties for low-income youth 


$46,305,452 


Broaden Educational Oppor- 
tunities for Disadvantaged 
Students 


QUASAR (Quantitative understanding: Amplifying 
Student Achievement and Reasoning): National demon- 
stration project in middle school mathematics 


$3,600,000 


Equity 2000: National demonstration project in 
mathematics, focusing on algebra and geometry 


$1,500,000 


Urban Partnership Program: Access to higher education 
for at-risk youth 


$870,800 


Knight 


Excellence in Education 


Collaboration between local schools and colleges to 
enhance school reform 


$2,400,000 


Special National Opportuni- 
ties in Education 


Cities in Schools 

Experiments in for-profit school management 


$2,650,921 


Lilly 

Endowment 

(IN) 7 


Education and Youth 


Teaching and learning; student access to higher educa- 
tion; public support for education including school 
reform and parent involvement; and education gover- 
nance 


$31,000,000 


MacArthur 


Education Policy 


Better Education through Informed Legislation Project: 
National Council of State Legislators 
- develop and implement education reform 


$175,000 


Development of Education 
Standards and Assessment of 
Measures 


New Standards Project: Student assessment that relies 
on nontraditional testing methods - National Center on 
Education and the Economy 


$3,000,000 


Research and evaluation on New Standards Project’s 
implementation and impact - Consortium for Policy 
Research in Education, Rutgers University 


$225,000 


Patterns of Thinking Project for research at Harvard 
University on thinking dispositions and creative 
thinking 


$300,000 


Marin 
Community 
Foundation: 
Buck Trust 7 


Education and Training 


School restructuring and redesign, drop out prevention, 
literacy and basic skills, and life-long learning 


$3,300,000 


James S. 
McDonnell 


Cognitive, behavioral, and 
social sciences research to 
improve curricula, learning, 
teaching and teacher training 


Cognitive Studies for Educational Practice Project: 
Collaboration to develop theory-based instructional 
prototypes that address K-12 instructional problems 


$7,400,000® 
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Initiatives of Foundations to Improve Education 1 


Foundation 


Designated Program Area 


Initiatives/Focus 


Funding 

Level 2 


Andrew W. 


Project 2061 


Nationwide project of the American Association for the 
Advancement of Science to improve scientific literacy 
or students 


$750,000 


Mellon 


Critical Thinking 


Learning Research and Development Center, Univer- 
sity of Pittsburgh to examine educational applications 
of research on varieties of reasoning 


$940,000 




Poverty 

- Children, Youth and 
Families 


Families and Early Childhood; 
Teenage Pregnancy Prevention; 
Youth Service and Employment 


$735,255 

$861,072 

$1,919,227 


Charles 
Stewart Mott 


- Education 


Historically and Predominantly Black Colleges 
and Universities 
School/Community Initiatives 
School-to-Work Transition 
Systems Redesign and Equity for activities 

involving education reform, collaborative models, 
and program evaluation 


$222,758 

$2,667,851 

$675,000 

$1,094,994 


Panasonic 


School Reform Partnership 
Program 


Reform efforts in nine urban school districts 


$594, 400 9 


William 
Penn (PA) 7 


Transitioning Adolescents to 
Adult Life 


Minorities in Higher Education 

Summer Youth Employment/Career Exploration 

Teenage Pregnancy Prevention 


$6,528,984 

$1,014,269 

$2,392,610 


Pew 

Charitable 

Trusts 


Education Restructuring 


New Standards Project: student assessment 
PATHS/PRISM, Philadelphia Partnership for 
Education: middle school restructuring 
Philadelphia Schools Collaborative 


$5,500,000 

$2,400,000 

$7,820,000 


Teachers and Teaching 


Improve undergraduate teaching and increase the num- 
ber of minority faculty in higher education 


$896,300 




Access and Success in Higher 
Education 


Increase number of disadvantaged students completing 
high school and transitioning to postsecondary educa- 
tion 


$7,760,000 


Rockefeller 

Foundation 


At-Risk Youth 


Equity 2000 


$2,000,000 


Rockefeller 

Brothers 


Education 


Minority recruitment and training programs, early 
childhood teacher education training programs, and 
support for special teacher education programs 


$854,070 


San Fran- 
cisco (CA) 7 


Education 

• i 


School reform, early childhood development, and re- 
cruitment of minorities into higher education 


$1,600,000 6 
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Initiatives of Foundations to Improve Education 1 


Foundation 


Designated Program Area 


Initiatives/Focus 


Funding 

Level 2 


Alfred P. 


Education in Science, Tech- 


Longitudinal study of career choice among secondary 


$2,999,905 


Sloan 


nology, and Management 


school students conducted by the University of Chicago 






Improve Elementary and Sec- 


Recruitment, Especially Minorities, to Teaching Profes- 


$22,890,825 




ondary Schools 


sional Development of Educators 


$16,394,600 






- Strengthening Teacher Preparation Programs for 


$1,1 13,500 s 






Middle Grades: Center for Early Adolescence 
Improve Capacity of Schools to Serve Children 


$6,679,700 




School/Community Collabo- 


Collaborations to Create New Learning Opportunities 


$4,278,242 




ration 


- Science Linkages in the Community 








American Association for the Advancement of Sci- 


$3, 672, 242 s 






ence 








School/Family Partnerships 


$3,389,400 


DeWitt 

Wallace 


Career, Service, and Educa- 


Integration of vocational and academic education, 


$12,463,000 


tion Opportunities for Youth 


workforce preparation 
- Career Beginnings Program 


$2,300,000'' 






School retention and transition to postsecondary 
education or full-time employment - Brandeis Uni- 
versity 








- Study of high school career academies - 


$2, 000, 000 s 






Manpower Demonstration Research Corporation 
- Expansion of High Schools that Work Program - 


$2, 775, 000 s 






Southern Regional Education Board 
- Work Force Skills Program: Research and technical 


$2, 290, 000 s 






assistance to restructure systems for educating and 
training workers - National Center for Education and 
the Economy 








- Improve ability to make choices about postsecondary 


$3, 3 00,000 s 






vocational training - Kansas State University 
- Reinforce academic skills - Public/Private Ventures 


$3 ,000,000 s 




At-Risk Youth 


Equity 2000: guidance counseling and evaluation com- 
ponents 


$5,925,000 


We in g art 


Education 


Preschool education, recruitment of minority teachers, 


$8,334,237 6 


(CA) 7 




application of technology to teaching 




Xerox 


Education 


Science and technology, recruitment of minorities in 
higher education, support of Institute for Research on 
Learning, and National Board of Professional Teaching 
Standards 


$7, 100,000 9 


TOTAL 






$476,724,070 
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Source: AIR 

1 . This listing includes foundations whose annual investment in education and related issue areas is over a million 
dollars annually or whose initiatives have otherwise garnered national attention. Twenty-eight foundations were 
identified, with total giving of about $462 million. 

2. 1992 awards unless otherwise noted. Information was developed through a variety of sources, and therefore, it 
remains to be determined whether totals represent annual or multiple year awards. 

3. 1994 awards. 

4. 1991 awards. 

5. This amount is reflected in the total award for this category. 

6. 1993 awards. 

7. Boundaries of limited service area.. 

8. 1987-1991. 

9. 1990 awards. 
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THE NEEDS, DEMAND FOR AND SUPPLY OF EDUCATIONAL RESEARCH, 
development AND DISSEMINATION 

This second of four background papers for the Research, Development and Dissemination 
System (RD&D system) Committee of the National Educational Research Policy and Priorities 
Board (NERPPB) moves on from the topic of the analytic mapping of the educational RD&D 
system considered in the first paper to a more detailed analysis of the nature of the research, 
development and dissemination work believed required and demanded versus that which is 
supplied. 

To consider this question in some manageable way, a basis for describing demand and 
supply must be selected. For the demand side, the first section considers why the conventional 
economic notions of demand do not fit the educational RD&D system, and then proceeds to 
consider and comment on four expressions of educational RD&D needs: the NERPPB statement 
in 1997, the National Research Council (NRC) review of 1992, the National Academy of 
Education (NAE) review of 1991, and the President's Committee of Advisors on Science and 
Technology (PC AST) report of 1997. 

For the supply side, manageability dictated the selection of a portion of the total supply. 

A part of the two largest sponsoring sectors -- the federal government and the foundations -- 
were identified as the basis for choice. In the case of the federal government, the Department of 
Education (ED) and particularly the Office of Educational Research and Improvement (OERI) has 
been selected. In the case of the foundations, an illustrative group -- not necessarily 
representative -- was selected on a basis described more fully in Section II below. Obligations 
corresponding roughly to the federal fiscal year of 1997 were used to obtain a modicum of 
consistency among suppliers. 

The authors (and we hope the readers) recognize the “rough and ready” character of the 
numbers. Precision in the numbers is lacking, estimation was required, and comparability of time 
periods is inexact. Better data simply do not exist. Nonetheless, the numbers are accurate 
enough to draw some clear comparisons between what expert judgment believes the system 
should be producing versus what, in fact, it is. 

I. Expressions of Demand and Need 

A. Market Demand 



A conventional economic approach to demand or need would look to the work being done 
as a reasonable expression of demand, assuming that there is a market for educational RD&D. 
While some aspects of a market exist such as competition for grants and contracts, educational 
RD&D is at most no ordinary market, and in some respects, no market at all. A few facts 
illustrate the point. 

A large part of the ultimate consumers -- the educational institutions — do not purchase 
the product, but rather have it purchased on their behalf by a variety of sponsors, many of whom 
have their own agendas about what product should be purchased. Further, education is a public 
good not subject to ordinary market rules (the knowledge gained from educational research is 
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often freely available). And still further, the sponsors in varying forms — the Congress, the federal 
executive branch, the foundations - specify who is eligible to compete, how much of total 
educational resources will be given to certain designated suppliers and how the work will be 
undertaken, as well as what work will be done. 

These conditions and practices make reliance on current spending patterns as an accurate 
picture of true demand unreliable. At best, current spending patterns are an interesting indicator 
of what is, or more likely, was a judgment about educational RD&D needs adjusted by sponsor 
perceived amendments to meet current conditions. Given the unreliability of what is as an 
expression of true demand, one is then left with reliance on expert judgment for a clearer 
expression of needs. 

We now turn to four such expert judgments which have directly or partially addressed the 
question of educational RD&D needs during this decade. It is noted that this is a “long-running” 
topic over four decades, but we have stayed with more recent expressions by recognized 
collective groups. 

B. Expert Judgment — The NERPPB Report of 1997 

Consistent with statutory requirements established in 1992, the NERPPB in collaboration 
with the Assistant Secretary of OERI issued its first report in 1997 under the title of Building 
Knowledge for a Nation of Learners. As is the case with all four expressions of need reviewed 
here, this report begins with a recitation of the strong need to do a more effective job of 
education, particularly from early childhood through the secondary level, because of the sharp 
upward shift in hard and soft skills required for satisfactory employment and a participatory life in 
our society. Among the four, this statement of the problem is couched in the least dire terms and 
is the least critical of the existing system. 

This report then affirms a relationship between appropriately focused and well done 
research and the improvement of the educational system. Good research is described as high 
quality work, steeped in the realities of its ultimate destination in schools, and desirably developed 
through learning communities and with recognition of the life-long nature of the educational 
enterprise (pp 11-15). 

The report then turns to a discussion of an agenda for establishing priorities for 
educational research. The intent of the priorities are to “...provide a keener focus...” and to set a 
“...framework for the development of new practical applications of research ....” Yet the report 
states that the Assistant Secretary and the Board “...refrained from ranking the priorities.” And 
further: “None of these priorities can stand alone.” (pp 17-18) Later in the document, 
researchers are advised that “...priorities are not prescriptive, but are meant to suggest areas of 
research that hold promise... .” (p 80) Seven priorities are then identified and discussed: 

• Improving learning and development in early childhood; 

• Improving curriculum, instruction, assessment and student learning at all levels; 
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• Ensuring effective teaching through improved teacher preparation and professional 
development; 

• Strengthening schools, particularly middle and high schools; 

• Supporting schools to prepare diverse student populations effectively; 

• Promoting learning in informal and formal settings; and 

• Understanding the changing requirements for adult competence. 

It is to be noted that these priorities are stated in consumer and institutional outcome terms rather 
than in research objective terms for the most part. Further, it is difficult to imagine any 
educational research work that has been left out. 

The report ends with two brief chapters on “putting priorities to work” and “powerful 
questions.” With respect to the first, the concentration is on the roles of the many stakeholders 
and their processes rather than a strategic plan for the work to be undertaken. The discussion of 
powerful questions is a call for their production rather than a list of what they are. 

C. Expert Judgment — The NRC Report of 1992 

At the request of the OERI in advance of the 1992 reauthorization hearings, the NRC 
undertook a broad review leading to the production of Research and Education Reform: Roles 
for the Office of Educational Research and Improvement in 1992. Though more broadly scoped 
than this paper (its findings are relevant to a later paper as well), this NRC report deals directly 
with educational RD&D needs as seen by its expert panel. 

Its problem statement is far more wide-ranging, detailed and acerbic than the previous 
paper's. It identifies educational research nationally as a desperately under-funded enterprise 
attributable to: (1) the practice-oriented nature of teacher education and subsequent teacher 
incentives and professional lifestyle; (2) a national predilection for quick fixes; and (3) poor 
quality research (pp 19-20). It describes the need to “raise the bar” in educational performance in 
more detail, encompassing basic academic knowledge, conceptual understanding, and the capacity 
for problem-solving (pp 9-12). And it identifies three flaws in thinking about the role of 
educational research: that all research is suitable for practice application; that the flow from basic 
research to widespread application is linear; and that research by itself can lead to reform. 

The critique of OERI is equally strong. The NRC report starts by noting the huge scope 
of the mission accompanied by a paltry level of funding. In its general critique of under-funding, 
it is particularly critical of short-changing field-initiated work and basic research, noting in the 
latter case the NRC complaint about under-funding basic research as early as 1977. It charac- 
terizes the agency, beyond its control, mired in controversy and politics, and overindulging in 
short-term projects and diffuse agendas. It dislikes the governance structure, found very little 
coordination and cooperation with others plus a spotty record in quality assurance and cumulative 
learning. Only with respect to sustained efforts and linkage to practice did OERI exceed their 
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reputation, according to the report, and in both cases, they could do better. With respect to 
linkage, the report interestingly noted that textbook publishers are rarely consulted (pp 107-134). 

Turning from the critique of then current practice and conditions, the NRC panel first 
addresses some broad concepts and practices appropriate to educational research. These center 
on four key values on which educational research should focus: (1) expanding the understanding 
of the fundamental aspects of human development, learning, teaching, schools and context; 

(2) pointing the way to effective curriculum, instruction and school organization; (3) providing a 
sound basis to sort sound innovation and policy from fads; and (4) assessing the status and 
progress of education systems (p 16). The panel explicitly recognized the complex nature of 
conducting and applying educational research, and suggests the development of learning com- 
munities, being tried in other fields. They also urge the learning of lessons from other fields with 
respect to field-initiated research and broad-based, coordinated R&D. 

The NRC report then expands its more general statements of research needs with more 
specific identification of educational RD&D priorities, supported with illustrations (pp 19-53): 

• The importance of cognitive science and its application to mathematics and reading; 

• Curriculum development and improved teaching which translate cognitive science to 
the real world of teaching and learning; 

• School restructuring; and 

• Monitoring the state of public education. 

The report closes this priorities review with a discussion of the importance of involving the 
Congress in the results of the research. The report concludes by reiterating the research values 
noted above, calling attention again to the importance of field-initiated work, though cautioning 
against the dominance of any one mechanism or any single discipline. In its recommendations 
related to the content and quantity of educational R&D (pp 135-172), the NRC panel calls for: 

• OERI leadership in expanding fundamental knowledge, promoting excellence and 
equity, and monitoring the state of education; 

• A balanced R&D portfolio with expansion of field-initiated research, basic research 
and sustained (longer term) work; and 

• Significantly increased staff and resources. The panel added that if such additional 
resources were not forthcoming, the OERI mission should be shrunk to fit available 
resources (emphasis added). 
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D. Expert Judgment -- The NAE Report of 1991 

Funded by The Carnegie Corporation of New York, The National Academy of Education 
assembled a distinguished steering committee for a project on funding priorities for educational 
research. The result was a report in 1991 entitled Research and the Renewal of Education. 

The principle problem, as seen in this report is succinctly stated: “The research base is 
under-funded, limited in scope, and lacks connection to what happens in classrooms. Research 
studies tend to be short-term and conducted in isolation.” (p 6) In explicating this summary, the 
report details not only the inadequacy of total educational R&D resources, but also a serious 
deficiency in field-initiated work. With respect to foundation funding, it criticizes a tendency 
toward “action projects” and the move from research to demonstration with much of the 
remaining research directed to evaluation and monitoring of small scale projects. The report 
states that foundations devote less than 4% of total giving to R&D, and less than 15% of 
educational grants to R&D. The report complains that R&D has come to mean “replication and 
dissemination” rather than research and development. 

The report’s data and findings, based importantly on a survey among stakeholders (though 
no summary is provided), adds to the concerns identified above an additional list of issues 
concerning the organization of the research work (pp 21-34): 

• Comprehensive effective strategies are lacking; 

• Patterns of support are episodic and unstable; 

• Studies tend to be short-term, small scale and not interconnected; 

• Public funding is of insufficient “critical mass”; 

• Public funding is subject to unhelpful funding allocation rules and changing direction; 
and 

• Opportunities for coordinating field-initiated work and theory building are too few. 

In the steering committee view, all of these deficiencies result in fragmented and theoretically 
diffuse R&D, slow to respond to needs, and too small, short-term and without inclusion of 
context. 

In prescribing solutions to the problems, the NAE report takes a much more process- 
oriented approach than the NRC panel review. This approach begins with several cross-cutting 
themes which call upon researchers to: illuminate what could be, as well as examine what is; 
recognize that learning occurs everywhere, not just in school; adopt a broad focus beyond a par- 
ticular school level; and recognize that more money will come as the result of better quality work 
rather than more of the same (pp 35-43). The report then goes on to identify “special oppor- 
tunities” (pp 45-60). These include a combination of process and substantive ideas, such as: 

• The cultivation of active learning for all citizens; 
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• The development of a new form of assessment that will improve instruction, not just 
measure student performance; 

• More research on educational opportunities and teaching approaches for minorities; 

• More research on the social organization of schooling; 

• Stronger connections between research and teaching; and 

• A long list of other ideas including study of: teacher education in life cycle terms, 
rationalized and standardized patterns of curriculum and their impact on teachers; 
ethnographic work on class-rooms; teaching higher order skills; technologic 
integration; how teachers do new things; teacher learning; and teacher acquisition 
and retention. 

The NAE report characterizes its recommendations as going beyond “what” needs to be 
studied to “how” education research can be strengthened (pp 8-11). The recommendations 
encompass five major components as follows: 

• More funding for educational R&D, specifically increasing the total from less than 
0.5% to 4-6% of total educational spending. 

• An increase in consensus-building and quality control, including study of the 
establishment of a National Panel of Reviewers. 

• Stronger organization of the research including more field-initiated work and a more 
systematic approach that addresses: cooperative and integrated research goals across 
agencies, longer federal commitments to research and experimentation, renewed 
federal role in large scale studies, more and larger longitudinal data sets, equitable 
resource distribution among performers, priorities related to pressing problems with 
probable high research pay-offs, clear incentives for researchers and practitioners, 
incentives for state and district participation, a quid pro quo of more money in return 
for longer-term, better coordinated projects, and long-term investment in emerging 
fields and new scholars. 

• The development of incentives for new researchers (by foundations and 
corporations). 

• Stronger linking of research to practitioners. 

E. Expert Judgment — The PCAST Report of 1997 

The Panel on Educational Technology, organized in the spring of 1995 under the auspices 
of the President's Committee of Advisors on Science and Technology, undertook a review of the 
application of several technologies to the K-12 education system. This review was conducted 
through briefings and discussions with knowledgeable stakeholders as well as a non-exhaustive 
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examination of the literature. The result is a Report to the President on the Use of Technology to 
Strengthen K-12 Education in the United States, dated March 1997. 

While addressed to a particular dimension of educational activities, the review grounded 
itself in comprehensive educational policy problems, and addressed educational research broadly 
as well. As with all other reviews considered here, the central educational problem seen in this 
report is “raising the bar” on the number and quality of high school and postsecondary graduates 
in order to meet the labor force and modem societal demands in the United States. 

A number of the findings of the review provide some interesting insights into the issue 
under consideration and judgments about educational research: 

• The panel asserts a quality control problem in educational research related to both 
methodology and politics. 

• They find foundation preferences for action-oriented projects as did the NAE 
review. 

• “Early stage” research on important work such as software development is unlikely 
to be done by the private sector because of the “free rider” problem. 

• Professional development for teachers is both critical and expensive. Each hour of 
professional development time in the US will add about $4-5 billion to educational 
costs in the nation. 

• Pre-service training on technology utilization in 22 states is of poor quality. 

• The nation now spends about $3. 5-4.0 billion annually on educational technology 
(1.3%), and should be spending somewhere in the $6-28 billion range. 

• Meta-evaluations appear to validate the efficacy of technology with some distressing 
specifics. For example, higher utilization of technology is inversely related to 
student needs, notwithstanding the fact that the disadvantaged students show the 
largest educational gains from effective utilization. 

• A “constructionist” approach to curriculum in which the teacher functions as a 
facilitator, appears to be the most effective and consistent with technology 
applications. 

The PC AST report contains six major recommendations (pp 7-10) as follows: 

• Focus on learning with technology, not about technology. Both are important, but 
the former is critical. 

• Emphasize content and pedagogy, and not just hardware. 
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• Give special attention to professional development. Professional development 
should be 30%, not 15% of a technology budget. 

• Engage in realistic budgeting. About 5% of the total K-12 budget should go for 
technology (about $13 billion annually nationally), not the current 1.3%. 

• Ensure equitable, universal access. Toward this end, Title I technology spending 
should stay at or above current levels. 

• Initiate a program of experimental research, both in general and in technology. R&D 
should increase from its present 0. 1% to 0.5%, or $1.5 billion annually. The 
program should include: (1) basic research; (2) “early-stage” research (new forms of 
software, content and technology-enabling pedagogy): and (3) rigorous large scale 
experimentation. 

F. Summary Analysis of Needs 

The expert judgments about the driving problem for the educational system and the 
importance of high quality and relevant RD&D to its solution is monotonously consistent among 
the reviews, differing only in their intensity. The NAE and the NRC reviews were the most 
extensive and explicit; and the evidence since those reviews in NAEP, TIMMS and analyses such 
as the Mumane and Levy work cited in the first paper serve to underscore the seriousness of the 
problem. Indeed, the seriousness with which one views this central problem is perhaps crucial for 
conclusions drawn about remedies. 



Essentially all of the reviews suggest an educational reform framework through which to 
review and frame improvements of educational RD&D. And there are some elements of 
consensus about directions which should be taken: 



• All agree that the total effort is under-funded substantially, with basic research being 
particularly undernourished. 

• All agree that the methodological foundations and research quality needs to be 
substantially improved, some think as a prerequisite to larger funding. Further, there 
is clarity about the need to properly assess micro-level research before declaring it 
ready for “prime time.” What are the appropriate standards of evidence for success? 

• All agree that a complete diet of small, short-term projects in a constant state of flux 
is a losing proposition, and that long-term, large scale experimentation and research- 
laden demonstrations are required both to learn and scale up. 

• Reviews earlier in the decade were consistent with the need for more field-initiated 
research, but facts discussed in Section II and some internal tensions discussed 
below make this finding less clear now. 



• All four earnestly talked about setting priorities; but to the extent that priorities 
means making choices, one refrained, two tabled some reasonably specific notions 
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and then talked about “balance,” something for every discipline or processes, and 
only one “hard-charged” the priorities question, albeit about only one part of the 
enterprise. One of the middle ground reviews, to its credit, made clear that failure to 
provide more resources should result in a trimmed OERI mission, though it did not 
venture how to do it. 

• All agreed that the translation of research to practice and practitioners is difficult and 
crucial, and suggested a variety of ways to involve the practitioners from the outset. 
Few, however, had much to say about the resources or resource allocations to 
dissemination and technical assistance. They did have strong views about 
professional development and the availability of good data, particularly longitudinal 
data. 

There is much less clarity and, presumably, agreement about other matters and some 
internal tensions among some of the points being made in most or all of the reports. Perhaps the 
most striking to one coming fresh to the issues is the apparent disinclination to deal directly with 
the choices about the substance of the work, in contrast with the processes by which such choices 
are made by others, the division of RD&D by the type of research, development or dissemination, 
or the allocation of resources among established performers (labs, centers, field-initiated, etc.). 
The latter receive the bulk of attention (or even Congressional specification); and while all are 
important, one could argue that the discussion of research priorities might well start with the 
questions: what new knowledge is important or critical to solution of the most critical educational 
problems? and what are the most productive research opportunities available? 

For the authors of this analysis, the most consistent answers from thoughtful and informed 
observers center on the need to increase the learning performance for those for whom it does not 
come easily. This should come through the expansion and application of the growing knowledge 
base from cognitive science to the class-rooms and the teachers and students who live there, 
bringing different experiences, skills and attitudes which will affect their needs and paths to 
progress. While there are almost countless other questions and issues which may be productively 
researched, if we fail with the central questions, we are unlikely to be content with the results. If 
this is correct, then it provides an additional way to examine priorities and needs which can be 
applied in our next section. 

Three other matters in reciting problems in the current RD&D system and prescribing 
solutions which deserve more clarity include the following: 

• A number of the reviews make brief or hesitant reference to the use of learning 
communities or learning organizations. While not extensively tried or conceptually 
and operationally standardized, the learning organizations concept provides a far 
more participatory and interactive knowledge building approach than more 
traditional evaluations. Such learning organizations may provide education with a 
more satisfying approach to the inclusion of practitioners in the knowledge building 
process than other techniques now being tried or advocated. 

• All of the reports refer adversely to politics in the educational research process, and 
suggest directly or otherwise the elimination of such intrusion (usually while 
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recommending a series of measures such as increased funding requiring action by the 
political process). While the structural remedies usually recommended to preclude 
adverse intrusion will be discussed in a subsequent paper, it is important here to be 
clear about the role of politics — good and bad — in the RD&D process. All 
sponsors, including elected political ones, have a legitimate say in establishing the 
objectives and making use of the results of the resources they dispense. What is 
objectionable is decision-making unrelated to merit or underlying need. Most 
successful public R&D programs have reached good results not by excluding politics 
altogether, but rather by inclusion in effective and appropriate ways. 

Finally, the statements of need reviewed here (and elsewhere) are often less clear 
than they need to be about strategies to move the RD&D system from wherever it is 
to the desired destination. With anything of the size and history of the educational 
RD&D system, changing the status quo — particularly significantly — as 
contemplated in these reviews requires phased deliberative action. The failure to 
provide a sensible implementing strategy is likely to insure continuation of the status 
quo. In creating a strategy, it is very useful to have a “triggering” event; and such a 
possible event exists in the yet to be defined research initiative in the President’s 
FY1999 budget ($50 million in OERI and $25 million in NSF). 



As noted above, the research priorities statements reviewed here also contain internal 
tensions among the points that they make as well as matters left unclear and ambiguous. Two, in 
particular, are important to identify and consider: 



• The first tension focuses on a sound concept of balance, perhaps accompanied by a 
disinclination to stir up disputes among colleagues, and the need to create priorities 
and some sense of direction. In this tension, balance becomes a device to avoid 
difficult, but necessary choices (someone is going to make them in the end anyway, 
perhaps someone with less information and basis for judgment). A priorities 
statement without choices provides neither the guidance to promote change and 
focus nor the basis for judging whether the statement has any impact. This is not to 
suggest that balance is an undesirable concept for the RD&D portfolio, but rather to 
suggest that it is an unsatisfactory “cop out” as a substitute for priorities. 



The second tension, both related and more difficult, focuses on the twin notions of 
participatory processes and field-initiated research as against the twin notions of 
national agendas and RD&D program coordinated coherence. Educational research 
agendas and priorities reviews, including those discussed here, routinely champion 
all four ideas without discussing how these inherent tensions are to be harmonized or 
resolved; so routinely they are not resolved to the detriment of all four notions. This 
generic problem is made more difficult by the continuing national fealty to the 
desirability of keeping the federal government out of educational curricular matters. 
This adds the challenge of how to be national without being federal, an issue with 
which we are struggling in professional teaching standards and also testing with 
apparently quite different levels of success. All that we know about such tensions 
suggests that clarity of the problem to be solved and an inclusive process to 
harmonize the tensions are important to effective resolution. 
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II. Selective Illustrations of Supply 

A. Selection Methodology 

In this paper, we focus on the Department of Education (ED) and foundations, as 
illustrative of the main funders of educational research, development, and dissemination 
(RD&D). Information on ED funding was obtained from several different sources including 
ED’s budget and OERI Institute summaries of research projects. 

In order to determine which foundations to focus on as the biggest funders of educational 
RD&D, we first identified those foundations with substantial grant-making in the education area. 
Next, we had discussions with experts to determine which of these foundations were most likely 
to be conducting work in education RD&D. We then examined the foundations’ annual reports 
to further identify their spending and to classify it into various categories. Last, we spoke to 
education program officers in the foundations to confirm our classification of their education 
spending and to clear up anything that was not spelled out in sufficient detail in their annual 
reports. Our levels of education research grant-making by foundations differ from those cited in 
the report from the National Academy of Education for several reasons including different time 
period (we look at 1996 and they look at 1989) and different definitions of education research. 

B. U.S. Department of Education 

Figure 1 shows the distributions of 1997 Department of Education RD&D obligations. 
This chart shows that demonstration makes up the largest component of these obligations (over 
40 percent). The next largest areas of spending are applied research, and dissemination and 
technical assistance. Development comprises only 5.3 percent of obligations and basic research 
comprises less than one percent of obligations. 

Table 1 shows that basic research spending by ED was approximately $2.9 million in 
1997 or three tenths of one percent of ED spending on RD&D. Approximately two-thirds of this 
spending was by the OERI Institutes (mostly in the areas of curriculum, achievement and 
assessment, and early childhood education) and roughly one-third of basic research spending was 
by the National Board of Professional Teaching Standards (NBPTS). Field-initiated research 
represents 26 percent of the Institutes’ total budgets. In addition, it is noted that the special 
education applied research budget exceeds that of OERI, and substantial demonstration and 
larger scale applied research are in the program agencies rather than OERI. 

Applied research, evaluation, and development accounted for nearly thirty-five percent of 
ED spending on RD&D. In 1997, approximately $240 million were spent on applied research 
and evaluation, and $45 million were spent on development. OERI Institutes accounted for 
almost $46 million of the applied research and evaluation total, and $6 million of the 
development total. Most of this spending was done by R&D centers. The regional labs account 
for a relative small portion of this funding, with only $7.7 million spent on applied research and 
evaluation, and $12.7 million spent on development. ORAD’s spending accounts for a large 
fraction of the demonstration spending. Most of this demonstration funding involves impact 
evaluations, but a large portion does not. 
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Figure 1 

Distribution of 1997 
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Table 1 

Department of Education 1997 RD&D Obligations 
by Knowledge-Building Category by Organization 
(Obligations $ in thousands) 



Organization 


Basic 

Research 


Applied 
Research & 
Evaluation 


Develop- 

ment 


Subtotal 
R&D & 
Evaluation 


Demon- 

stration 


TA, 

Dissem- 
ination & 
Other 


Total 


OERI Institutes 


1,817 


45,840 


5,840 


53,497 


0 


502 


53,999 


R&D Ctrs 


(1,784) 


(26,766) 


(3,140) 


(31,690) 


(0) 


(0) 


(31,690) 


Field initiated 


(0) 


(13,982) 


(0) 


(13,982) 


(0) 


(0) 


(13,982) 


Other 


(33) 


(5,091) 


(2,700) 


(7,824) 


(0) 


(502) 


(8,327) 


Regional Labs 


0 


7,650 


12,750 


20,400 


0 


30,600 


51,000 


National Dissemination 


0 


0 


0 


0 


0 


18,567 


18,567 


System & Bd 
















ORAD 


1,084 


16,997 


2,149 


20,230 


171,245 


0 


191,475 


NBPTS 


(1,000) 


(2,000) 


(2,000) 


(5,000) 


(0) 


(0) 


(5,000) 


G&T Center 


(84) 


(1,267) 


(149) 


(1,500) 


(0) 


(0) 


(1,500) 


All other 
















w/ impact eval. 


(0) 


(0) 


(0) 


(0) 


(114,340) 


(0) 


(114,340) 


wo/impact eval. 


(0) 


(13,730) 


(0) 


(13,730) 


(56,905) 


(0) 


(70,635) 


NCES 


0 


0 


0 


0 


0 


82,617 


82,617 


Assessment 


(0) 


(0) 


(0) 


(0) 


(0) 


(32,617) 


(32,617) 


Other 


(0) 


(0) 


(0) 


(0) 


(0) 


(50,000) 


(50,000) 


Elementary, Secondary 


0 


1,000 


2,000 


3,000 


152,979 


25,554 


181,533 


Special Education 


0 


90,676 


7,000 


97,676 


0 


49,872 


147,548 


Rehab. & Disability 


0 


41,000 


8,000 


49,000 


18,942 


0 


67,942 


Vocational & Adult 


0 


5,179 


5,000 


10,179 


0 


4,998 


15,177 


Higher Ed. 


0 


1,000 


2,000 


3,000 


0 


0 


3,000 


PES Evaluation 


0 


24,268 


0 


24,268 


0 


0 


24,268 


Other Misc. 


0 


6,000 


0 


6,000 


0 


0 


6,000 


TOTAL 


2,901 


239,610 


44,739 


287,250 


343,166 


212,710 


843,126 


PERCENTAGE 


0.3 


28.4 


5.3 


34.0 


40.7 


25.2 


100.0 



Table 2 and Figure 2 break down ED’s applied research and development spending into 
several categories. The largest fraction of this spending was in the area of at risk K-12 students, 
with nearly 30 percent of ED’s spending in this area. The other dominant areas of spending were 
comprehensive reform, with an over 14 percent share; early childhood with a 12 percent share; 
and postsecondary, with an over 10 percent share. 
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Postsecondary 9,054 

Centers (5,000) 
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Policy 7,574 

Centers (4,400) 

Field- Initiated (1,697) 
Other (1,477) 


ECI 8,028 

Centers (3,344) 

Field- Initiated (2,792) 
Other (1,862) 
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Other (1,853) 
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12.0 


6.0 


00 


00 


94.0 
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00 
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28.9 


36.0 


00 


00 


00 


74.1 


20.4 
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00 
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Percentage Distribution of 1997 National Institute R&D 
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Demonstration without research accounts for approximately forty percent of ED’s RD&D 
spending. Non-research demonstration spending was approximately $343 million in 1997. ORAD 
accounts for $171 million of this funding, the Office of Elementary and Secondary Education 
accounts for $153 million, and the remainder is funded by the Office of Rehabilitation and 
Disability. 

Dissemination and technical assistance accounts for approximately another one-fourth of 
ED spending on RD&D; approximately $213 million in 1997. NCES accounted for nearly $83 
million of this spending and the Office of Special Education, nearly $50 million. Other major 
funders of dissemination and technical assistance were the regional labs and the Office of 
Elementary and Secondary Education. 

We were unable to divide spending by regional laboratories into these topical categories, 
due to the nature of their operation, which includes considerable work across categories. How- 
ever, we have categorized spending by the regional laboratories into various types of research on 
the basis of their descriptions. Table 3 shows that the largest concentration of funding by regional 
laboratories is in the areas of language and cultural diversity, as three regional laboratories, with 
funding totaling over $14.5 million, concentrated on this topic. Other specialty areas include 
urban education; rural education; early childhood education; curriculum, learning, and instruction; 
educational technology; assessment and accountability; and school change processes. 

C. Selected Foundations 

Figure 3 displays the percentage distribution of 1996 foundation grant-making. 
Comprehensive reform and dissemination activities are by far the largest areas of spending by 
foundations, accounting for over 50 percent of foundations grant-making. The next largest areas 
for research funding are postsecondary education and curriculum research and development. The 
remaining areas of research each account for less than 5 percent of total foundation grant-making. 

Table 4 shows foundation grant-making in greater detail. The selected substantial funders 
of education RD&D in 1996 were the Annenberg Foundation, and the Ford Foundation, both 
with over $20 million of funding. Below are some descriptions of individual foundation research 
concentrations. 

The Annenberg Foundation RD&D funding concentrates on comprehensive school 
reform. In 1993 the foundation announced a $500 million challenge grant to create and replicate 
school reform programs around the country. While most of this funding is for actual program 
implementation, some funds have been set aside for research, evaluation, and assessment. 
Additionally, part of this funding has been used for the Annenberg Research Institute, which 
focuses exclusively on research and development. 

The Carnegie Corporation’s education program has four sub-programs: early childhood, 
young adolescent, education reform, and general education. Their research spans several areas 
including curriculum, achievement and assessment, early childhood education, at risk K-12, 
comprehensive reform, and professional development. One example of their work is a program 
which provides technical assistance to schools in fifteen states to help them implement education 
reforms, and to measure the impacts of the reforms. Another example is a project which studies 
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Table 3 

Spending by Regional Laboratories 
(In Thousands) 



Lab 


1997S 


Research 


Specialty 


Northeastern 

Region 


6,055 


Assessment; Parent Involvement; Policy; Professional Development and 
Leadership; Equity; Use of Technology 


Language and 

Cultural 

Diversity 


Mid-Atlantic 

Region 


5,227 


Systemic Reform Led by Local Schools; Organizing Urban Communities 
for Systemic Educational Reform; The Urban Fellows Program; Fostering 
Educational Resilience and Learning Success; Integrative Analysis of the 
Knowledge Base 


Urban Education 


Appalachian 

Region 


4,149 


School Readiness; Academic Achievement; School -To- Work 
Opportunities; Designing Professional Development; Scaling Up Internet 
Use 


Rural Education 


Southeastern 

Region 


5,612 


Improving School Capacity to Monitor Progress Toward Reform Goals; 
Improving High Schools’ Self-Assessment Capabilities; Scaling Up 
Efforts to Ground Teachers’ Professional Growth and Ability to Make 
Major Changes in their Approach to Instruction; Studying the Impact of 
Individual State and Region-Wide Policies and Practices on Schools and 
Students (particularly those in disadvantaged districts and communities); 
Working with Teenage Parents 


Early Childhood 
Education 


Midwestern 

Region 


6,637 


Curriculum, Instruction, and Assessment (Technical Domain); Human 
Development, Learning, and Motivation (Personal Domain); 
Organizational Learning and Development (Organizational Domain); 
Systems Integration and Systemic Change 


Curriculum, 
Learning, and 
Instruction 


Southwest 

Region 


5,526 


Enhancing Family and Community Involvement in Education; Addressing 
Diversity; Language and Culture; Aligning and Supporting Policy 
Development; Changing the Organization and Management of Schooling 


Language and 

Cultural 

Diversity 


Central 

Region 


4,194 


The Learning Program; The Teaching Program; School Development for 
Learning; Leadership for Learning; School and Family Partnerships for 
Learning; Readiness for Learning: Early Childhood Education; 
Community Development: Schools at the Center 


Educational 

Technology 


Western 

Region 


5,465 


Whole School Reform; Language and Cultural Diversity; Early 
Intervention, Care and Education; School -to- Work Transition; Teacher 
Assessment and Certification; Development and Use of Scoring Rubrics 
for State Performance-Based Assessment Systems 


Assessment and 
Accountability 


Northwestern 

Region 


5,169 


Assessment and Accountability; Early Childhood Education; Rural 
Education; School Change Process 


School Change 
Processes 


Pacific Region 


2,965 


Characteristics of Effective Language and Culturally Diverse Schools and 
Teachers; Effective Alternative Assessment in Multi-Cultural/ Language 
School Settings; Effective Strategies for Engaging Language and 
Culturally Diverse Populations, Parents, and Communities; Linguistically 
and Culturally-Based Strategies for Fostering Student Motivation and 
Learning 


Language and 

Cultural 

Diversity 
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Figure 3 

Percentage Distribution of 1996 Selected Foundation Grant-making 




Table 4 

Annual Research, Development, and Dissemination Grant Making By Foundations - 1996 

($ In Thousands) 
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how to improve learning and development for students aged three to ten. A third example is the 
development of a voluntary teacher assessment system. 

The DeWitt Wallace Reader’s Digest Fund focuses on three programs in education: 
school libraries, professional development, and the recruitment of nontraditional teaching 
candidates for low income districts. Only a small portion of their funding is used for research and 
dissemination. 

The Ford Foundation is the largest funder of education RD&D, with nearly $20.3 million 
in 1996. The Ford Foundation funds research in the areas of curriculum, achievement and 
assessment, at risk K-12 education, comprehensive reform, postsecondary education, and 
professional development. Ford’s greatest areas of spending are in curriculum, comprehensive 
reform, and dissemination. Some examples of their work are developing area studies programs 
and evaluating public school reform initiatives. 

The IBM Foundation changed its traditional pattern of computer support and hardware 
grants to more proactive partnerships with schools in educational grant-making in comprehensive 
reform in the early 1990's. Under the Reinventing Education Program, in 1994 10 grants totaling 
$25 million were made for 3-5 years to help finance comprehensive reform with a variety of 
different models followed by 12 two-year grants totaling $10 million to pursue the more 
promising model in the first round. An external evaluation is underway with an interim process 
evaluation of the first 10 grantees already issued by the Center for Children and Technology. 

The John D. and Catherine T. MacArthur Foundation funds education research in the 
areas of achievement and assessment, comprehensive reform, postsecondary education, life-long 
and adult learning, and professional development. Their greatest amount of funding is in the 
technical assistance and dissemination area. Some examples of research projects they have funded 
are an evaluation of the New American Schools reform initiative, and supporting projects that 
enhance the professional development of teachers. 

The Andrew W. Mellon Foundation is a major funder of postsecondary education 
research. Some examples of their research activities include a study of improving the quality and 
effectiveness of graduate education in ten research universities, testing how electronic materials 
can improve teaching and reduce costs, and a survey which provides data on college outcomes. 

The James S. McDonnell Foundation concentrates on basic research in the area of 
cognitive science and on curriculum reform. In the area of basic research some projects they have 
funded include “Functional Organization of Human Ventral Visual Areas” and “The Neural Basis 
of Skill Learning Using FMRI.” Their applied work in the curriculum area includes the 
development, implementation, and assessment of instructional materials for teaching science to 
students in grades 1 to 5; and developing and studying how software can help students to learn 
and express themselves. 

The Pew Charitable Trusts fund education research in the areas of curriculum, 
achievement and assessment, and postsecondary education. They also fund a great deal of 
dissemination. Some examples of their research and dissemination include an evaluation of 
stand ards-based reform in seven school districts, development and dissemination of models of 
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performance funding for public colleges and universities, and the provision of technical assistance 
to urban school districts for implementation of standards-based reform. 

The Rockefeller Foundation funds research in the areas of curriculum, achievement and 
assessment, comprehensive reform, postsecondary education, and professional development. 
Additionally, they fund large projects which have dissemination components. Some of their 
research projects include a study of the Equity 2000 precollege intervention program, evaluation 
of the subject knowledge of teacher candidates, and an analysis of the Comer School 
Development Program in San Diego city schools. Some examples of their dissemination activities 
are acting to increase the impact of the report of the National Commission on Teaching and 
America’s Future through support of public outreach and implementation of its findings, and 
expanding the number of schools using Comer’s school improvement approach. 

The Spencer Foundation funds research programs to support educational researchers at 
different stages of their professional careers. They provide grants to graduate students, 
postdoctoral researchers, and faculty members to support work that shows promise for improving 
educational practice. Areas of research covered by these grants include basic research in 
cognitive science, and applied research in the areas of curriculum, achievement and assessment, 
early childhood education, at risk K-12 education, comprehensive reform, postsecondary 
education, life-long and adult learning, professional development, and other areas. They also fund 
a small amount of dissemination. 

The AT&T Foundation was contacted, but not included in the tabulations because of the 
newness of its program. In 1996, it completely revamped its educational program from traditional 
employee matching gifts to higher education institutions and grants to business related science and 
engineering programs to a proactive partnership with school districts in comprehensive reforms 
centered on technology They established a $50 million 5-year program starting in the last half of 
1996, not unlike the IBM program with a technology focus. 

D. Summary 

While both the Department of Education and selected foundations concentrate on applied 
rather than basic research, their emphases within applied research are somewhat different. Both 
Ed and the foundations focus on comprehensive reform and postsecondary education. ED also 
emphasizes at risk K-12 education and early childhood education. The foundations have a greater 
focus on achievement and assessment and curriculum research and development. Areas of applied 
research that appear to be de-emphasized by both sectors are school finance, professional 
development, and life-long and adult learning. 

Comparing the current distributions, practices and conditions in the Department of 
Education and the selected foundations with the expert judgements, there are both changes from 
the time of the system reviews at the beginning of the decade described in Section I and suggested 
changes where either little has changed or matters have gotten some worse. Specifically: 

• The admonition to increase field-initiated work, helped no doubt by Congressional 
allocation formulas, has resulted in a substantially larger share, recognizing that the 
total amounts remain small. 
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• Serious work is being done to establish more rigorous standards for micro research 
and experimentation labelled promising or exemplary. The use of peer review is 
more frequent, and more demonstrations have evaluation or research components. 

• Stakeholder, including practitioner, participation continues in traditional and 
sometimes token and ineffective ways. 

• Most everything else remains largely unchanged. Total resources have grown very 
little, particularly in the research part of OERI, but no one has yet taken seriously the 
NRC review admonition to trim the OERI mission in the absence of more dollar and 
staff resources. 

• Basic research has grown not at all, and both the National Institute for Child Health 
and Human Development (NICHD) and the Office of Naval Research (ONR) have 
more significant cognitive science programs than ED. The research to expand and 
translate that work to the classroom remains to be addressed in the federal sector. 

• The federal applied research and development is still dominated by small, short-term 
projects. Any large scale federal demonstrations are directed to marginal change and 
financed outside of OERI. Large scale experimentation and demonstrations are 
financed outside the federal government. With the splintering of an already “sub- 
critical” sized staff into five components tied up with allocation rules, it is unclear 
how the institutes could mount an aggressive experimentation program. 

• The foundations, as the reviewers alleged, do appear in part to be favoring action- 
oriented projects, though not all of them. They are also paying more attention to 
curriculum development and professional development. 

• There is no sign yet of a coherent, integrated national educational research agenda 
even at the broadest levels. The research agenda is highly decentralized to the 
centers and field-initiated performers. And there is as yet, no clear plan to develop a 
process to weld it together, though the Department's adoption of the concept of an 
analytic agenda is a beginning. 

IH. Implications for the Congressional Report 

The implications of this analysis of the need versus the supply of educational RD&D are 
substantial for the field and potentially significant for the Board's report to the Congress. There 
remain significant and damaging shortfalls between perceived needs by a succession of the 
reviewers and the supply responses. Resources are so far short of perceived needs that they 
constrain options and reasonable responses to other deficiencies. The educational basic research 
component is growing up outside the Department and the link to practice remains unaddressed. 
The federal government appears to be getting out of the large scale experimentation and 
demonstration field with respect to comprehensive reform. The intermediation and dissemination 
mechanisms appear outmoded for at least comprehensive reform support, and perhaps expensive 
for other purposes. Whatever conceptual merit the Institutes may have had, at their present 
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staffing and resource levels, they are likely marching themselves into irrelevance. And past 
remedial prescriptions have not sold well. 

This somewhat gloomy diagnosis leaves the Board with some challenging questions: 

• If the Board is persuaded that the diagnosis is correct, is it prepared to take on the 
battle again? If so, how is it going to adjust the prescription so that it has a better 
chance of success? Is it, for example, prepared to use the FY1999 budget initiative 
as a way to set some priorities, restructure processes and capacities, and set forth a 
strategy that can build confidence among funders and stakeholders? 

• Does it make any sense in restating basic research needs to seek to recapture the 
cognitive science agenda within OERI, or instead, build links to the work where it 
now exists and the agenda for the translation into applied form? 

• Can the work going on to implement GEPRA, the analytic agenda, and support of 
the new initiatives be harnessed together to begin the development of a strategic 
RD&D plan and system which creates a non-federal, national agenda driven by the 
urgent need to improve K-12 performance? 

As we complete the remaining papers and prepare for the March meeting, we will hone 
these questions and potential responses for the Board’s consideration. 
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This third of four background papers for the Research, Development and Dissemination 
System (RD&D system) Committee of the National Educational Research Policy and Priorities 
Board (NERPPB) proceeds from the prior analytic mapping and the demand and supply topics to 
consideration of alternative styles in the conduct of federal RD&D programs. After consultation 
with the Committee, two examples were selected: the cognitive science program of the Office of 
Naval Research (ONR) and the National Institutes of Health (NIH) with particular attention to 
the learning disabilities and reading program of the National Institute of Child Health and 
Human Development (NICHD). 

The methodology used to prepare this paper included the collection of written informa- 
tion about each of these programs followed by conversations with knowledgeable individuals 
within or hierarchically over the concerned programs. This analysis also relies on the consid- 
erable experience of the author with both Defense and domestic RD&D programs, though not as 
a cognitive scientist. 

The paper will discuss the ONR and then the NIH/NICHD programs, followed by a com- 
parison with the Office of Educational Research and Improvement (OERI) and the implications 
of the analysis for the Board's Congressional report. 

I. Cognitive and Neural Science and Technology in ONR 

The cognitive science and technology program resides in the broader context of a quite 
formal and structured RD&D program process within the Department of Defense and a long 
history of quality research in ONR, dating back to World War II. In a structure first adopted in 
1962 for the planning, budgeting and management of RD&D that has changed remarkably little 
since, the Department and its component parts consider RDT&E (research, development, test and 
engineering) through six major categories starting with basic research (known as 6.1), 
exploratory development (6.2), then advanced development (6.3), and on toward engineering, 
prototyping, testing and operational manufacturing. Although primarily designed for hardware, 
the structure is applied to behavioral end educational research as well. All of the ONR work of 
interest is done in the 6.1, 6.2 and 6.3 categories. 

The actors, the processes, the documentation and the decision criteria for each of the 
categories are formally spelled out. The work proposed and undertaken is always linked directly 
or indirectly to important missions or knowledge-building priorities. Even for basic research, 
potential naval relevance or relationships to core competencies for the Navy need to be 
established and understood. Attached is an extract from the Navy manual which describes the 
process. Notwithstanding the formalism and bureaucratic structure, the ONR has maintained a 
consistent reputation for very high quality work, no doubt through the use of the seasoned 
judgment of its technical staff. 
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One of the clear causes of its positive reputation lies in the long history of very strong 
internal technical capabilities within ONR, which in many areas conducts its own work as well 
sponsoring and overseeing the work done by extramural resources. This characteristic as well as 
the need for secrecy in some elements of the total ONR program leads to other distinctive 
characteristics of the RD&D style. External peer review does not exist in the award of grants and 
contracts, relying instead on review by internal expertise. Relatively few formal efforts are made 
to create incentives for high quality or new researchers beyond the incentives created by the 
solicitations for grants and contracts and the money they provide for awardees. The technical 
staff do, however, cruise the technical conferences as well as the literature with an eye to 
emerging talent, and a portion of the grants are set aside for young investigators, 1-5 years 
beyond their terminal degrees (usually turns out to be those 5 years out with burgeoning publi- 
cation records). And the dissemination activity in ONR is quite traditional: internal briefings by 
researchers or supervising staffs to interested Navy staff to summarize or explain technical 
reports, and journal publication to meet the interests of non-Navy consumers. 

The cognitive and neural science and technology program is large and diverse, and the 
concentration here is on the work most related to education. Even at this narrowed definition, the 
budgets are substantial (an estimate is under development). The project grant sizes in the 6. 1 and 
more modest 6.2 efforts will run in the $50,000-150,000 range, while the project budgets for the 
more ambitious of the 6.2 efforts and all of 6.3 work will exceed $1 million in total and not 
infrequently $1 million per year over 3-5 years. The program included in 1996 more than 1 10 
performers of which 63% were universities, many among the leading national research 
universities. The program ranges over five areas of training-related research: 

• Computational models of the mechanisms of human learning and performance (6.1); 

• Effective instructional strategies, emphasizing tutorial instruction (6. 1/6.2); 

• Specialized areas of computer science needed for training applications (6. 1/6.2); 

• Exemplary demonstration applications (6.2/63); and 

• Authoring tools for affordability (6.2). 

The very strong orientation to research goals and underlying mission can be seen in the 
way program managers articulate an integrated set of research questions: 

• “What is the precise nature of the expertise that is the goal of training? 

• What are the internal learning processes that transform the novice into the expert? 

• How do we feed those learning processes the experience needed to optimize 
progress? (This is the problem of instructional strategies.) 
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• How do the answers to these questions differ for different major domains of 
expertise?” (from an internal briefing chart of April 23, 1 997) 

The program is also firmly anchored in eventual practical application in three different types of 
training delivery options (embedded training, shipboard training and schoolhouse training) with 
demonstrable cost effective results. For example, an intelligent computer-assisted instruction 
(ICAI) model developed by John Anderson of Carnegie Mellon was reporting 33% reductions in 
instructional time and a typical one standard deviation improvement in achievement. This 
tutoring model was evaluated in a controlled experiment with 9th grade algebra word problems 
which showed moderate improvement against SAT and Iowa test problems and large 
improvements on problem solving and multiple representation questions on proportion of correct 
answers (experimental n = 470 and control n = 120). These results were then correlated with an 
evaluation of an Air Force avionics tutoring model and other Navy console operation 
experiments. 

The ONR program reflects a strong sustained commitment in training technology with 
computer assisted instruction and computer-based training in the 1960s, with interactive video in 
the 1970s, with artificially intelligent tutoring starting as early as 1969 and becoming a major 
emphasis in the 1980s, and with virtual reality in the 1990s. The program is continuously 
dealing with two key aspects: first, the development of psychologically valid models of 
knowledge which can be learned by humans and expressed in computational form; and second, 
the development of sophisticated instructional apparatus which simulates problems and provides 
effective instructional strategies, assessment methods and trainee interface. And the results must 
always meet cost effectiveness tests. For example, the high-end ICAI models are rich in detail (if 
this, then that) and student prompting (e.g., cutting off faulty reasoning paths), and thus complex 
and expensive. The question then becomes whether the high end models produce significant 
improvement in human performance and are cost effective relative more rudimentary and generic 
models plus individual reasoning capacity. 

In this program, ONR considers its open research issues important to achievement of the 
two standard deviations improvement in learning effectiveness than it desires to be found in 
pursuit of: true natural language interaction, effective instructional strategy options, comparative 
value of various cognitive models, the added value of sophisticated tutoring strategies, the 
effective design and integration of multimedia, and the effective use of virtual reality. Some of 
these issues are illustrated in the prior paragraph. These issues pose not only technical problems, 
but substantial costs, probably requiring some guessing about potential payoffs. While not the 
subject of this paper, it is interesting to note that ONR staff suggesteci that the biggest and most 
readily transferable target of their work would be to vocational education where they perceive 
little work to be now underway. 
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II. The Learning Disabilities and Reading Program of NICHD/NIH 

Prior to discussing the NIH program most closely connected to educational research, it is 
appropriate to set the context more broadly in health research and the operation of the NIH. 
Health research like the national security function is more generously endowed than education by 
huge amounts. The annual NIH budget exceeds $13 billion, complemented by large 
expenditures by foundations and private health institutions producing health products and 
services, in the latter category most particularly pharmaceutical companies. It is important to 
note at this point that the structure of the health enterprise is such that there are substantial 
incentives for significant private investment in health productions, since the potential returns are 
attractive and patent protected. Notwithstanding these other sources of sponsorship for health 
RD&D, the federal government is both a dominant actor and regulator, without an ideological 
overlay about the appropriateness of the federal role. 

All of this funding should not be assumed to relieve health RD&D from issues of 
priorities, planning and budget constraints. Like ONR, the health RD&D system operates within 
the context of a formal process established by the Department of Health and Human Services 
(HHS). At the broadest levels, the Department sets forth both substantive and process guidance, 
now in the form of strategic plans required by the Government Performance and Results Act 
(GPRA). In its first plan in September 1997, HHS identified four principles which provide 
structure to its research investment strategy: priori ty for basic research, priority to investigator- 
initiated research, reliance on peer review for quality assurance, and commitment to a broad and 
qualified research base. On those four principles, it then imposes a set of six broad strategic 
objectives. Attached is the most recent research goal statement. 

As discussed in the second paper, HHS has evolved a process which they believe 
effectively balances the tensions between capitalizing on the creative and stimulating virtues of 
investigator-driven projects with the needs of national program coherence at the institute and 
programmatic levels. The mechanisms include the work of institute director advisory groups and 
grant review councils which together with periodic statements of critical research questions help 
to achieve consensus and coherence of the aggregate agendas ( based on personal communica- 
tions with Dr. Susanne Stoiber, Deputy Assistant Secretary of Planning and Evaluation, HHS 
and Dr. G. Reid Lyon, Chief, Child Development and Behavior Branch, NICHD). 

The NIH has, of course, developed through time into one of the world's premier R&D 
organizations. That development, particularly since the 1960s has been marked by a strong and 
growing public support during which time the Congressional action more often than not 
increased Administration requests for appropriations. In the late 1960s, NIH Director Shannon 
cultivated a close and supportive relationship with the Congress which has withstood the test of 
time and periodic Presidential efforts to impose budget discipline. The NIH has and continues to 
face problems such as: how to deal with rare diseases and conditions, how to increase innovative 
research, and how to encourage new researchers against the “old boy network” tendency of peer 
review, but it has rarely had to contend with out-of-control political micro-management or 
ideologically-based external specification of the details of research agenda or performer 
specifications. 
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Turning from the broad structure and processes of the NIH and its institutes to the 
specifics of the Learning Disabilities program and its dominant component -- reading research, it 
is large and diverse. Following its selection as the result of legislation in 1985 as the interagency 
center for work on learning disabilities, NICHD began to rapidly expand a program dating back 
to 1963 with efforts directed to finding the causes and potential remedies for reading difficulties 
among children and adults. From FY1985 through FY1997, NICHD has invested almost $143 
million in learning disabilities -- and predominantly — reading research. The annual levels in 
FY1995-1997 are respectively $14.1, $15.8 and $21.8 million. 

The work is now conducted in 1 8 different sites organized under 5 learning disabilities 
centers, 5 multi-disciplinary program projects for the study of language development and 
disorders and dyslexia, and 3 large research programs to study treatment and intervention 
approaches for learning disabilities in oral language, reading and written language (several 
programs have more than one site). (Lyon, “Progress and Promise in Research in Learning 
Disabilities, "Learning Disabilities, Vol. 8, No. 1, Winter 1997) The basic and small scale work 
is carried out in university, clinical and school setting, but there is also substantial large scale 
assessment and remediation work underway in school settings. 

The larger scale school site work is occurring in Houston, Texas, in Colorado, the D.C. 
schools, Albany, New York and Seattle. Houston has been underway for 7 years with a 
longitudinally followed data base of 4000, D.C. schools with 800 per year over five years, Seattle 
with 600 per year and the others with small but significant samples. The target groups are 
disabled or at risk students, meaning that the population is not all disabled. The research design 
is not and cannot be experimental random selection, but it is quasi-experimental with differing 
treatment groups, longitudinal tracking, and common assessment and protocols for each of the 
three research categories across all sites. The program is reporting that effective treatments are 
able to raise 95% of the treatment group up to national reading standards. The program has been 
working almost exclusively in reading, but is beginning to venture into mathematics. While the 
program is working with large populations and in schools, it has not yet directly experienced the 
problems of district scale-up, but the Director states that those issues will be confronted through 
Robert Slavin’s comprehensive reform work. 

The program is structured around seven different types of disabilities; listening, speaking, 
basic reading skills, reading comprehension skills, written expression, arithmetic calculation 
skills, and mathematic reasoning skills. They are assessed to be different from each other in 
etiology, developmental course and response to instruction. The research strategy seeks to: 

• Identify etiolologies; 

• Develop early predictors; 

• Map the developmental courses of the disabilities; 
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• Identify commonly co-occurring disorders and secondary social and behavioral 
characteristics; and 

• Identify and assess the efficacy of treatments. 

So far, the program believes it is learning several important things. First, it estimates that 
approximately 33% of all children are disabled or at high risk with respect to reading, and 
perhaps as many as 60% may be at risk for less than optimum reading skills. Reading 
deficiencies in turn produce high risks for serious problems in educational development, motiva- 
tion, self-esteem, and a host of subsequent problems. They find little difference between males 
and females, conventional wisdom notwithstanding, and believe that there are reliable ways to 
assess phonological deficits in late kindergarten and first grade. For those with reading 
difficulties at 9 years of age, 74% will continue to have reading problems throughout their school 
tenure. 

The findings of the NICHD work has embroiled their research in the political battle 
between phonics and whole language, with the phonic enthusiasts in the Congress believing 
incorrectly that the findings supports the phonics approach exclusively. In fact, the research does 
support the criticality of phoneme awareness and the development of the alphabetic principle, but 
also the importance of an additional set of skills involving the construction of meaning from text 
which invokes aspects of the beliefs of those in the whole language camp. Many students with 
proactive and skilled parents in reading preparation will proceed through the development of 
phoneme awareness and alphabetic skills with lightening like speed, making prolonged drilling 
over extended periods of time an educational demotivator. For those who have difficulties with 
those initial phases however, it is a great mistake to proceed further until those skills are secured. 

Without them, the researchers believe, reading competence will not be achieved. 

The program’s leadership advocates expansion and extension to deepen its understanding 
of the causes and instructional strategies which will address yet better difficulties in reading. 
With respect to assessment, the program director testified before Congress on July 10, 1997 
(House Committee on Education and the Workforce) that a phonemic assessment tool for young 
children had been developed which in 15 minutes at $10-15 a case can predict reading deficiency 
with about 92% accuracy. As a part of a list of needed remedial action in his testimony, the 
director called attention to the inadequate preparation of teachers of reading and the importance 
of competency based training and board certification. 
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III. Comparisons with OERI 

The RD&D processes in a Departmental context are newer and less well articulated for 
education than for ONR or NICHD. The new Strategic Plan for 1998-2002 of September 30, 
1997 in response to GPRA requirements is the Department’s second effort. Selected pages from 
that document bearing on the RD&D process are attached. Like its counterparts in HHS and the 
Navy, it sets forth broad goals and objectives, and it does a particularly commendable job of 
relating R&D effort to mission objectives (the ED plan is generally ranked in the top three 
among domestic agencies). It is unlike the comparative illustrations in two undesirable ways: 

• It is much clearer about the mission and application goals than it is about the 
research goals and priorities, though it should be noted that the HHS-wide document 
is quite general too. In HHS, the scope of NIH is so large that much of the detail is 
left for derivative documents. 

• There is a strong sense in the ONR and NIH cases that the science base is firmly 
rooted, and that there is a clear sense of direction and cumulative learning. In the 
comparative cases, the growing knowledge base is a powerful determinant of both 
future research and operational actions, giving knowledge-building a crucial role in 
future improvement. There is no such comprehensive sense for educational RD&D. 

As known from other analyses, the OERI agenda is not yet appropriately linked into the Depart- 
mental process, though initial steps are underway (see Morrill & Weiss, Talent, Tensions and 
Transition —An Organizational Analysis of the Planning and Evaluation Service, February 
1997). HHS and ONR have this linkage far better in hand. 

Although repetitive with the two earlier papers, one is again struck in this comparative 
analysis with huge differential in resources among the three agencies which inevitably impacts 
the style adopted. But it is more than just staff and dollar resources; it also goes to the issue of 
whether the education RD&D program is an endless series of small applied research projects 
unrelated to an evolving critical set of knowledge bases or a cohesive agenda of cumulative 
knowledge-building. On this criterion, the comparative agencies are more well-developed. 

The related issue of how to resolve the tensions between the values of field initiated and 
participatory processes and the need for national agenda and coherence is handled quite 
differently by the two comparative agencies. ONR has a strong hierarchical process which 
assures the coherence, perhaps with sufficient openness at the detail level to encourage 
innovation. In any event, it is not a useful model for OERI. HHS attacks the problem through 
the institute director advisory groups, grant review councils and periodic articulation of critical 
research questions in combination with strong reliance on field-initiated research. HHS staff 
report an ability to achieve effective balance with these processes. NIH is a relevant model. 
While OERI has made progress on increasing the field-initiated component of its program 
(virtually all of the institute programs), it has yet to develop the coordinated coherence in any 
meaningful way. 
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With respect to quality assurance, ONR makes clear that there is more than one way to 
achieve it. Careful consideration of the context, however, leads one to understand that peer 
review would not fit the ONR context very well, while NIH could not survive without it in spite 
of a very strong in-house staff. And peer review is the appropriate route for OERI, which it is 
now trying to implement. It is important to recognize where peer review processes are appro- 
priate and where they are not. In this context, peer review is understood to mean not just the 
collective judgment of experts, but the effective power to decide as opposed to advise. With this 
meaning, peer review is effective in ranking proposals and evaluating project performance. It is 
not, appropriately, elsewhere used to set institutional goals or strategic objectives which is 
properly left to institutional leaders and sponsors with technical advice from expert groups. 

The NIH and ONR have largely positive and proactive relationships with the elected 
political decision-makers in their institutional lives in which they maintain the initiative in 
consensus-building. OERI has no such situation, and is clearly less proactive. Indeed, the effort 
appears to have been more on keeping executive branch and Congressional political actors out of 
the RD&D program rather than seeking their constructive participation. The alternative strategy 
uniformly works better and will be revisited in the roles and structure paper (paper #4). 

The critique of OERI as having insufficient sustained effort on critical topics discussed in 
the second paper is underscored with the long-run commitments and cumulative learning in the 
ONR and NIH programs. This condition complements the finding of excessive small, short-term 
projects in OERI noted above. Further, OERI is like NIH and ONR in having both close and 
more distant organizational access to large scale experimentation and demonstration resources, 
but is considerably less well linked to such activity. 

Finally, the comparison of OERI and ONR/NICHD basic research is important on both 
conceptual and substantive grounds. On conceptual grounds, the comparison provides confirma- 
tion of the judgment that OERI sponsors far too little basic research. On substantive grounds, 
NICHD and ONR sponsor cognitive science which, particularly in the NICHD case, would be 
appropriate for OERI. Given these commitments, it would appear foolish to attempt to replicate 
that effort in OERI. Instead, it would seem wiser and more effective to link and expand it to 
meet OERI needs, and contemplate the initiation of complementary basic and applied work in the 
translation of the cognitive science knowledge base more broadly into classroom settings. 
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IV. Implications for the Congressional Report 

The implications of the comparative analysis is to suggest that models do exist for 
effective RD&D systems which it is important to notice are built only over time. These systems 
are to some degree — usually defined as “critical mass” -- self-contained in that they have the 
stature and resources to do the work as well as to plan and oversee it. Indeed, they are unlikely 
to be able to do the second job well, if they don’t have the recognized stature to do the first. 

With respect to the planning and oversight, there are some clear and useful examples of 
how to strengthen the process and performance of the system as a whole, and the Department and 
OERI have initiated steps which can be pushed forward, so long as progress on the first set of 
issues identified above are also pursued aggressively. 

Finally, the route to a more appropriate balance for basic and applied research would 
appear to be to build on and link to existing efforts focused heavily on strengthening the needed 
knowledge base in learning. 



Attachments: 

1 . Navy RD&A Management Guide 

2. HHS Strategic Plan 

3. ED Strategic Plan 
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Production and Deployment, the fielding of the 
system to provide mission capability needs of the 
Navy and Marine Corps. The double arrows at 
each link emphasize that the RD&A process is not 
a simple linear progression, but rather involves 
much iteration and feedback. 

Test and Evaluation (T&E) is a vital constant, 
occurring in every' phase of development and 
production. Formal T&E. governed by a carefully 
developed plan, begins in the Demonstration and 
Validation phase (inferred to infru ma’.ly as 6.3B, 
see 2. 1.1.3). Successful RD&A programs are 
invariably characterized by (1) effective use of 
T&E to manage risk and (2) early and effective 
communication and coordination among 
developers and prospective manufacturers and 
users of the system under development. T&E is a 
complex subject, which is discussed in detail in 
Chapter 7 and Appendix G ol this Guide. 

2.1.1 Categories of RD&A Effort. For planning, 
funding and review purposes, the Defense 
RDT&E Program is structured in six categories. 
These categories often are referred to by the 
numbers of the categories under the DOD 
Programming System. The six categories arc 
described below. 



Ref: DON Budget Guidance Manual; 
(NAVCOMPT 7102.2) 



2. 1.1.1 Research. Budget Category 6.1, 
Research, includes all effort of scientific study and 
experimentation directed towards increasing 
knowledge and understanding in broad fields 
directly related to long-term DON needs. 
Research is conducted to ensure that both 
cutting-edge scientific discoveries and the general 
store of scientific knowledge are optimally utilized 
in the development of superior Naval equipment, 
strategies, and tactics. The 6. 1 program is a major 
source of basic and applied research effort in 
electrical engineering, materials science, applied 
mathematics, and other disciplines (sec C3. 1 .2) of 
importance to the DON. 
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2. 1.1. 2 Exploratory Development. Budget 
Category 6.2. Exploratory Development, includes 
effort directed toward the solution of specific 
Naval problems, short of major development 
projects. The role of the exploratory development 
program is to ensure that, as technological, 
advances appear, they are investigated for possible 
development/exploration to determine applica- 
bility to future Naval programs. This type of effort 
may vary from fairly fundamental applied research 
to sopb'Sticated bread-hoard hardware. 

2. 1.1. 3 Advanced Development. Budget 
Categoty 6.3, Advanced Development, includes 
ail projects which are characterized by the 
development of hardware for experimental test. 
The prime result of this type of effort is proof of 
design. At the core of the Advanced Development 
program is the imperative to develop and make 
available to the Fleet new and advanced 
technologies that will ensure the long-term 
superiority of U.S. forces. Advanced Develop- 
ment is further broken down into 6.3A and 6.3B 
effort. 6.3A projects are not related to specific 
ship or aircraft applications and arc mostly 
developed as Advanced Technology Demon- 
strations (ATDs). Only highest risk/highest payoff 
efforts are selected as 6.3A projects. Many 
programs bypass this risk projection/risk 
assessment step and proceed directly into 6.3B 
programs which develop technologies intended for 
application on specific systems, e.g., ships or 
aircraft. 

2.1.1. 4 Engineering and Manufacturing 
Development. Budget Category 6.4, Engineering 
Development, includes programs that are typically 
in engineering and manufacturing development but 
have not yet received production approval. 

2.1. 1.5 Management and Support. Budget 
Category 6.5, Management and Support, includes 
support of installations or operations required for 
general research and development use. Included 
would be test ranges, military construction, 
maintenance support of laboratories, operations 
and maintenance of test aircraft and ships, and 
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Studies and analyses in support of the R&D 
program. 

2. 1.1. 6 Operational Systems Development. 
"6.6." Operational Systems Development, 
includes those projects still in Engineering and 
Manufacturing Development, but which have 
received approval for production or for which 
production funds have been included in the DOD 
budget submission for the budget or subsequent 
fiscal year. All work in this area is identified by 
major line item projects that appear as "RDT&E 
Costs of Weapon System Elements” in other 
programs, e.g.. Program l. Strategic Forces. 
Although Operational Systems Develop- 
ment is an official budget category, ”6.6" is a 
term used for convenience in reference and 
discussion. Thus, there is no program element 
number 6.6xxx. 

2.2 PLANNING FOR RD&A 

For the categories of effort described above, 
except 6.5, this chapter will provide overviews of 
the following: 

• Principal Participants. Provide a listing of 
the principal individuals and organizations 
involved in planning. 

• Documentation. Describe the top-level 
documents used in planning and their ' 
purpose. 

• Process and Schedule. Where a process 
for the development of planning for a 
category has been established, provide a 
brief description of the process. Provide 
an insight into the liming of the planning 
activities. 

• Decision Criteria. Provide an overview of 
the decision criteria for each category. 

2.2.1 Budget Category 6.1, Research. 

2.2. 1.1 Principal participants. The Chief of 
Naval Research (CNR) is responsible to the 
Secretary of the Navy (SECNAV), through 
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ASN(RD&A), for the overall investment strategy 
for the Research program and for developing 
research and technology programs which 
effectively address future operational naval needs 
and capabilities identified by the Chief of Naval 
Operations (CNO). 

The Director. Science Directorate (ONR) is 
responsible to the CNR for managing the Research 
Program. 

The CNO is represented by the Director, Test 
& Evaluation and Technology Requirements 
(N09I), who acts as Resource Sponsor for 
planning, programming, and budgeting S&T and 
as representative of the user in establishing 
warfighting requirements for S&T. 

2.2.1.2 Documentation. There are two key 
documents used during the development of the 
annual Research program: 

(1) ONR Navy Research Investment Strategy 
which provides overall strategy for Naval 
research; and 

(2) ONR Naval Needs & Science 
Opportunities which provides specific direction 
for allocation of Navy research resources. 

2.2. 1.3 Process and schedule. Long-range 
naval objectives and requirements are integrated 
with promising scientific opportunities into an 
annual program plan. CNR and N091 conduct an 
annual process of evaluation, prioritization, and 
selection of submitted research proposals for 
approval by A$N(RD&A). 

2.2.1.4 Decision criteria. Decision criteria 
for approving 6.1 Research projects are: 

• Does the project maintain a broad 
sustaining, versatile program in all science 
areas of potential naval relevance? 

• Does the project emphasize core 
competencies in ocean sciences, advanced 
materials and information sciences to 
accelerate technology transition in DON 
high priority area? 

• Does the project maintain balance in the 
program? Note: The 1992 program 
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Invested 60% of funding in evolutionary 
research, 15% in; high risk/high payoff 
revolutionary effort,, and 25% in research 
closely associated with Fleet applications. 
Does the project train students in science 
and engineering disciplines '''crimpy 
' needed by the Navy in nurturing a strong 
■ and ; i responsive- -in-house laboratory 
research capability? 

•■'■•Does” the 7 ^project" accelerate transitions to 
- meet critical scientific ’gapsTn essential 
-'Fleet programs? - 

2.2.2 ; Budget Category >6.2, ^ Exploratory 
Development. 'i'-AC' f&'sg&iitepti.' 



bn ... 



,i . . * 



j • - >** • * s-- •> 



Ref: OCNR Instruction 3910.3 . .. - . .. . 

. ■ ■ . if •: .kj~ j ;.J0 

.>>' f 2^2.Z.i Principal pjaiiticipant&^ 

. .Navai ; Research..' (Cl^y;?^ 
j.SECNAV, through^^NQj^^ : 

investment strategy for Exploratory Development. 
.. for deYelopmg.raea^..a^ programs; 

which effectively address ^ future^research , and 
future aha capabilities 

. identified- by die’CNO?. 5 ^^ 

' The- .7Dire(^r7-^Ti^T 1 &.'? EvSuatioii^ ; and 
Technology (N091) &K'WR^ui^S{M^or'for* 
planning, programming,' and fiudgeliiiig^ 
as ' representative ' of the ' user in establishing 

warfighting requirements' fbFS&Tt ’ •• "* * 

i-.. -The Director, TechnologyDirectorate(ONR) 
is responsible • to the '-CNR -for -managing'.' the 
Exploratory Development program. He is assisted 
by the Science Directorata.which helps to identify 
high-leveraged opportunities. . .. 

2. 2.2. 2 Documentation. The Exploratory 
Development program consists of -a- set of Block 
Programs which ' are ‘‘ documented in : Block 
Program Plans; developed by NaVy Laboratories 
and R&D Centers. A' Block' Prbgram Plan is 
submitted for"eich"Blcick 'Program: - 'A Block 
Program ’ is ' an tntegfated 'group r of : technology 



projects with closely related applications and/or 
technical objectives. The Block Program Plan 
describes the program to be executed for the 
Execution .year and Program .. Objectives 
Mem orand um (POM) years.. ... .... .. t . 

: :c2.2.23 Process -..and :: ’schedule;: Jr. The 
Exploratory : Development program development 
integrates . -.the;. : planning, programming,, and 
budgeting processes. -The processes carried out on 
an annual. basis with each quarter of the fiscalyear 
emphasizing:. a: distinct portion, of the process as 
follows:;-' ■ f: 

... . “ First Quarter: Accountability. ‘Technol- 
s •-^ogy Directorate management reviews and 
... . .. assesses the previous and current years’ 

.. ....... programs. ;Jf . ... / .... . ... 

• Second Quarter: Strategic Pla nning . The 
-Investment and Mission Area Strategies 
are> developed by the ; Technology 

: Quarter: Execution Planning. .'The 

<3uid^<»6;feisued add the 
'•^•vt^fssrBlock Elans -fon the following fiscal-year:-. 

;■ ■>>“ ; are developed..;. ■ ■ 

..... Fourth Quarter: Block . Program Plan 
: ^.'^reviews;' 1 modification,' 'approval, and 

■ - r ‘ • :.>'>fundihgT :A l-^ ^docum^ •. . are 

* . ; promulgated’ by ’30 'September f " 

/Plosion .. cnteria^The-^decision 
. criteria., for. Exploratory Development, programs 
are: • . - . 

• Does the program Keip'to majntaxn Navy 
technological superiority and provide the 

' capability to counter hew_threat$? 

• ""Does the program provide technology 

opportunities that preserve the strategic 
Naval initiative and extend strategic 
• . 'flexibility? 

Does: -.- the;; . program improve:: : the-. 

effectiveness of the U.S. deterrent 

■ W h % .>1. 

^ Ddes-':'1 ihe' : program'’; presenPfsignificarit' 

' ~ f ''threats-' ta tT-S.. adversaries?- »- ■ "v--^ 
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* Does the program provide technology that 
reduces cost of acquisition and operation 
and maximizes system cost-effectiveness? 

2.2.3 Budget Category 6.3A, Advanced 
Technology Development (ATD)^'^ : '' ? . 

i ,2.2.34 Principal participants.The Chief of 
Naval . Research. .. (CNR) is.-, responsible -to 
.^^pN^Yri^9 u Sh/,ASN(Rp&A): for. the. overall 
investment strategy for Advanced. Technology 
.Development and for developing,, research and 
..technology programs -which, effectively address 
.future operational naval needs and capabilities 
identified by the CNO. .. 

The Director; ' Test & " Evaluation and 
Technology Requirements (N091) acts as 
Resource Sponsor for planning, programming, 
and budgeting S&T and as representative of t thie 
user in establishing warfighting requirements ' for . 

S&T. .. . I-* 14- 

The pirectorfTechnoiogy Directorate (ONR) 
jis re spoi^ie^to^ttte_CNR for managing jthe 
^Advanced Technology Development program: iHe 
is assisted bythe Science Directorate, which helps 
to identify high-leverage opportunities. \p.) 






2.2.3.2 Documentation.. - Advanced-,- 
; Technology Development proposals preparedby 
,Navv Laboratories; Warfare Centers and industr^ 
provide inputs for developing the 6.3A program. 

2.2.3. 3 Process and schedule. OPNAV 
XN091) establishes ATD requirements. Based on 
those requirements, CNR solicits proposed 
concepts, reviews the concepts, and submits its 
evaluation to N091. N091 reviews the concepts 
and submits its evaluation to ASN(RD&A). 

After ASN(RD&A) review and concurrence, 
CNR requests that the proposers develop detailed 
concept papers for selected concepts. • :i 

The CNR in conjunction with the ATD"' 
executing organizadons, evaluates the proposed 
concepts and develops a list of recommended ATD 
program for N091 to consider. 



Z.2.4.2 

The ATDs program is updated andueccssary 
Planning, Programming, : and Budgeting. System 
(PPBS) actions taken. 

. .. After PPBS and . .Congressional - approval , 
CNR is .authorized to proceed with the.. ATD 
execution,/- 

.... . ,2-2.3.4 Decision criteria. The design criteria 
for ari ATD are: '• :r -Z • ■' ■ 

*.?. Is itjeesponsive to pr ion ty-warflgh Ling 
requirements?;;^ .d -hex 

y :r. z it consistent with DDR&E.guidance and 

, ■ •' o-* ■ ' coordinated. iw ith -Project Reliance? (See 
k:wiRc^3.-L), mo » KO* ' 

Does it have medium to high risk but high 
'•.•.">:luci transition potential and is it scheduled tq be 
completed in 3 to 6 years? 

• Does it have strong OPNAV sponsorship? 

•.•<2:2.4 Budget Categories 6.3B, 6.4, and “6.6,” 
-vAdy^ced,-, , Engineering,, and Operational 
. -Sys^s^Deveio^en^ . . ... 

j -i; ^4^4 ^^ iid'pad...' pa^cIpantsB^ Program 
uUv e ~ Officfe rs : (PEOs) Direct Reporting 
Program Managers (DRPMs) , Systems Command 
Commanders /:::(SY, SCOMs) , and.;?:- 7 Program 

^MWagers-^Ms)- : are responsible tov the. Navy 
- Acquisition Executive (NAE) to plan and execute 

-the. programs,^-.. . . .......... 

• =-.'.v^8s6urce : Sponsors within OPNAV (Surface 
Warfare, .Submarine Warfare, a nd Air Warfare) 
are responsible. fqrr(lj representing die: warfighter 
in the acquisition proccss^and (2) represehtfhg't 
’ ’program iii tile PPBS; . , 






Ref: pop Directive' ' 5000 j 'P-Part 3;\ :■ 

^ pECNAV Instruction 5000.2. - ■■■■■■■■ ■ 

2.2.4.2 Documentation. Milestone reviews 
require rigorous assessments of a program’s status 
.. and plans for. the future. The information needs of 
the -milestond.decisron‘“authority and supporting 
staffs at each level, however, must.. be satisfied 
'without creating' ah undue burden on the Program 
Manager. Accordingly, the milestone . review 
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documentation concept, established by DOD 
Directive 5000.1. DOD Instruction 5000.2 and 
DOD O000.2-M. provides for stand-alone 
supporting documentation and two standardized 
in i oi mation displays, the Integrated Program 
Summary and the Integrated Program Assessment, 
as shown in Exhibit 2-2. DOD Instruction 5000.2. 
Parts 2 and 11, provides additional information on 
these documents, including who prepares, 
validates, and approves them. DOD 5000.2-M 
prescribes format and content. 

Stand-Alone Supporting Documentation. 
The purposes of the stand-alone supporting 
documentation such as the Cost and Operational 
Effectiveness Analysis (COEA), Test and 
Evaluation Master Plan (TEMP), Acquisition 



Program Baseline (APB), the Prosram and 
Independent Cost Estimates (PCE & ICE), and the 
Manpower Estimate Report, arc to meet the 
information needs of the milestone decision 
authority, Supporting staffs, and review forurns. 

Integrated Program Summary (IPS). The 
purpose of the IPS is to provide a succinct, 
integrated picture of the program's status, from the 
Program Manager's perspective, for use by the 
milestone decision authority, supporting staffs, 
and review forums. 

Integrated Program Assessment (IP A). The 
IPA is a critique of the TPS and summarizes the 
results of the independent assessments conducted 
by the supporting staff and review groups. It is a 



standalone documents 

- TEST & EVALUATION MASTER Pu\N* 

* PROGRAM COST ESTIMATE 

- WDEPENOENTCGST ESTIMATE* 

* COST 4 OPERATIONAL 
EFFECTIVENESS ANALYSIS 

* ACOUlSmON program 
8ASEUNE* 

* MANPOWER ESTIMATE REPORT* 
•waivers/reports* 




AMNEXES 



A. PROGRAM STRUCTURE 

B. PROGRAM LIFE-CYCLE COST 
ESTIMATE SUMMARY 

C. ACQUISITION STRATEGY REPORT* 

D. RISK ASSESSMENT 

E. ENVIRONMENTAL ANALYSIS* 

F. AFFORDABILITY ASSESSMENT 

a £2 S, PGrative opportunties 

.COCUMENT* 



INTEGRATED 
PROGRAM SUMMARY 



1. EXECUTION STATUS 



t 



2- THREAT MIGHUGHTS- 
SHORTFALLS OF EXISTING 
SYSTEMS 

3. ALTERNATIVES assessed & 
RESULTS 

4. MOST PROMISING ALTERNATIVE 
A RATIONALE 

a acquisition strategy 

a cost orivers a major 
TRADEOFFS 

7 * 385 assessment A Plans to 
REDUCE RISK 

a affordability of selected 

ALTERNATIVE 

a recommenoattons 



‘STATUTORILY IMPOSED REQUIREMENT 



INTEGRATED 
PROGRAM ASSESSMENT 



t. EXECUTION STATUS 

2. THREAT HK5HUGHTS- 
SHORTFALLS OF EXISTING 
SYSTEMS 

а. alternatives assessed & 

RESULTS 

4. MOST PROMISING ALTERNATIVE 
A RATIONALE 

5. ACQUISITION STRATEGY 

б. COST DRIVERS A MAJOR 
TRADE-OFFS 

7. RISK ASSESSMENTS A PLANS TO 
..REOUCE RISK 

8. AFFORDABILITY Of SELECTS 0 
alternative 

9. RECOMMENDATIONS 
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niajor issue-oriented document and provides (he 
basis for the milestone decision review agenda. 

DOD Instruction 5000.2, Pan 2 and 
Pan 11, Section C; DOD 5000.2- M 



2.2.4.3 Process and schedule. DOD 
Instruction 5000.2, Part 13. establishes the process 
for milestone reviews for Category ID by the 
Under Secretary of Defense for Acquisition once 
the Program Manager determines that the program 
has achieved all the objectives of the current 
acquisition phase and is ready to proceed into the 
next acquisition phase. Other acquisition 
categories, i.e., IC, n. III, and IV follow similar 
processes as established in SECNAV Instruction 
5000.2 and OPNAV Instruction 5000.42. 

Schedules for most programs have five 
milestone decision points and five phases during 
the acquisition process as illustrated in Exhibits 
2-3 and 2-4. Low risk programs may not need a 
Demonstration-Validation phase, in which case 
Milestones I and O may be combined. These 
provide the basis for comprehensive management 
and the progressive decision-making associated 
with all programs. 



Ref.: DOD Instruction 5000.2, Part 3 



2.3.1 

2 .2.4.4 Decision criteria. The decision 
criteria are based on rigorous, objective 
assessments of a program’s status and plans lot - 
managing risk during' the next phase and the 
remainder of the program. The acquisition 
strategy and associated contracting activities 
explicitly link milestone decision reviews to events 
and demonstrated accomplishments in 
development, testing, and initial production. 

Note: Chapter 1 ot this Guide, “The Acquisition 
Process — an Overview. " provides what, might be 
termed a “map’ of the official DOD acquisition 
process. The graphics are amplified by 
information on facing pages which identify the 
particular parts of DOD Directive 5000. 1 . DOD 
Instruction 5000.2, and DOD 5000.2-M that 
contain information on the documents, officials, 
and actions depicted in the graphics. 

2.3 SCIENCE AND TECHNOLOGY 
INITIATIVES 

The DON Science and Technology effort, 
which is managed by the Office of Naval Research, 
has been die subject of the recent Defense 
initiatives described below. 

2.3.1 Cooperation/Coordination in Science and 
Technology within DOD. In late 1989, the 
Deputy Secretary of Defense challenged the 
Services to create a new approach to Science and 




Exhibit 2-3 — Acquisition Milestones and Phases 
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STRENGTHEN THE NATION’S HEALTH 
SCIENCES RESEARCH ENTERPRISE AND 
ENHANCE ITS PRODUCTIVITY 



Improvements in health are grounded on knowledge acquired through research conducted 
and sponsored by the Department of Health and Human Services (HHS) and other 
entities, both public and private. In the scope and quality of the science it sponsors, HHS 
sets the pace for the world in medical, epidemiological, behavioral, and health services 
research. It does so through strong, sustained public support for health sciences research. 

The National Institutes of Health (NIH) plays a vital role in the nation’s medical research 
enterprise. NIH-sponsored research generates knowledge that leads to improvements in 
the health and quality of life of the American public. It also provides a continually 
expanding knowledge base for the development of commercial products by the 
pharmaceutical, medical device, and biotechnology industries and by other key 
components of the national medical research infrastructure. Through its support of 
research training, the NIH provides the nation with highly trained scientists who rise to 
leadership in publicly funded research activities and in the biotechnology and related 
industries. To a significant degree, future improvements in the health of the American 
people depend upon sustaining both the research infrastructure that has been developed 
through NIH support and the basic principles that have enabled NIH research investments 
to be highly productive. 

The Centers for Disease Control and Prevention (CDC) also conducts a strong program of 
epidemiological and population-based research to protect the public health and prevent 
and control disease, injury, and disability. 

Finally, the Department’s health services research plays a critical role by identifying what 
is most effective and cost-effective in day-to-day practice in community settings and by 
identifying the most efficient approaches for delivering and financing those services. 
Expansion in research investments across a broad front of scientific disciplines and 
operating divisions within the Department will do more than anything else to ensure 
improvements in health status and in the kinds and quality of services sponsored by the 
Department. 

The Department also enhances the productivity of the nation’s research enterprise through 
such means as international scientific cooperation and regulatory policies that encourage 
investments in research by the private sector. 

O 
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Four principles are central to the Department’s research investment strategy: 

Basic Research. First is the high priority accorded to basic research in the life sciences 
and fundamental methodological work in health services research. In the area of life 
sciences, one need look no farther than the history of the biotechnology industry to see the 
wisdom of this approach. HHS investments in basic research undergird epidemiological, 
clinical, and health services research. In the case of the last, they have laid a foundation 
for better administration and reimbursement in both the public and the private sectors. 

Investigator-Initiated Research. The second principle is the high priority accorded to 
sponsorship of investigator-initiated research. The Department traditionally has 
eschewed top-down direction for science and instead has relied primarily on individual 
scientists to propose and carry out specific research projects within the context of broad 
program goals and policy priorities enunciated by its agencies. As a result, HHS has been 
uniquely effective in harnessing the creative energies of scientists throughout the nation 
toward improving human health and well-being. 

Peer Review. Third is the reliance upon peer review to assess the quality of research 
proposals and outcomes. Determining the relative technical merits of competing research 
ideas is one of the most difficult tasks facing any research agency. The Department’s 
success year after year in directing investments to the most promising scientific 
opportunities and the most capable investigators stems largely from its commitment to 
seeking and heeding the advice of leading experts drawn from the pertinent scientific 
communities. 

Research Capacity/Infrastructure. Fourth is the Department’s commitment to 
sponsoring research in a wide variety of institutional settings and to encouraging a healthy 
research enterprise in the for-profit sector. Universities, not-for-profit research 
organizations, hospitals and other practice settings, small and large businesses, and 
government laboratories — each in its own way has provided a hospitable environment for 
important scientific initiatives. Also, in view of the unique role played by academic 
institutions, the Department will continue its efforts to help research-intensive universities 
remain strong. In particular, it will maintain its policy of paying a fair share of research 
costs and will collaborate with academic health centers to find ways to counter the 
destabi lizin g effects of managed care upon clinical research and the education of health 
professionals. Up-to-date and safe research facilities and equipment are essential to 
ensuring progress in medical research. The Department will continue its commitment to 
providing support for new and refurbished buildings and facilities for the conduct of 
medical research, as well as for the maintenance, renovation, and construction of facilities 
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for HHS intramural research. Finally, the Department will enhance the base of highly 
qualified scientific investigators. 

The Department reaffirms these principles as the core of its strategy to guide new 
investments to increase the nation’s knowledge base about health science and maintain its 
quality. HHS intends to apply them even more broadly in the years immediately ahead. 
To guide action and measure progress in achieving its goal, the Department has 
established the following strategic objectives. 



Advancing the understanding of 
fundamental life processes is 
essential for progress in improving 
health and combating disease and 
disability. New knowledge of the 
biological and behavioral 
processes that operate throughout 
the lifespan affords the most powerful means for understanding the course of disease and 
how it can be delayed, treated, cured, or ultimately prevented. 

Basic science studies matter at all levels of aggregation, from the materials we experience 
every day down to their most fundamental constituents. This progress leads to new 
scientific and technical knowledge and, years later, to innovative products and 
commercial profits. These advances have generated millions of high-skill, high-wage 
jobs and significantly improved the quality of life for Americans . 44 

The link between medical research and improving health is well illustrated by the vast 
new knowledge on the characteristics of various human genes that is accumulating almost 
daily. This research is providing many new concepts and tools for understanding the 
molecular mechanisms of various kinds of cancer and other chronic diseases; it is also 
opening the way to new avenues for prevention and therapy. 

Strategy for Achieving Objective 

HHS will advance its knowledge of biological and behavioral processes in these ways: 

■ sustaining its priority of investment in medical research; 

■ adhering to the investigator-initiated, peer-review model for selecting the highest 
quality research proposals; 



Strategic Objective 6.1 
Improve the Understanding of Normal 
and Abnormal Biological Processes and 
Behaviors 
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■ implementing decision-making mechanisms and policies to ensure that medical 
research is responsive to emerging health needs, scientific opportunities, and new 
technologies; and 

■ promoting technology transfer through partnerships with industry that enhance the 
federal capacity for medical research and facilitate the flow of new technology. 
(This task will be accomplished through the development and implementation of 
policies, procedures, and guidelines that facilitate patenting, licensing, and 
cooperative research.) 



Measures of Success 



■ The body of knowledge that has resulted in an improvement to the understanding 
of normal/abnormal biological processes and behavior. 



New Legislation and Regulations 



None. 



Key External Factors 

Progress in achieving this objective will be influenced by increasingly constrained 
resources for the scientific endeavor and the largely unpredictable nature of scientific 
discovery and advancement. 



Medical research is yielding 
knowledge that can be translated 
into new and improved methods 
for detecting, diagnosing, and 
treating disease. New insights into 
normal disease processes will also 
provide the knowledge for developing preventive measures, which are the ultimate 
manifestation of improvements in health. 

The importance of this research is summed up by a 1996 National Science and 
Technology Council report. Technology in the National Interest . 43 The report recognizes 
the contributions of medical research when it states, “[mjedical research supported by the 
National Institutes of Health has led to many discoveries that have improved both the 
health and quality of life of the American people. This is the NIH’s foremost goal, but 
medical research also yields technological and economic benefits.” 

In addition, research at CDC is contributing to the understanding of pathogens and 
immunology and the basis for the prevention of the transmission of diseases. 

Strategy for Achieving Objective 

The strategy for achieving this objective parallels that for Objective 6.1. 



Strategic Objective 6.2 
Improve the Prevention, Diagnosis, and 
Treatment of Disease and Disability 



Measures of Success 

Changes in the prevention or delay of the onset of disease and disability. 

Changes in the diagnosis of disease and disability. 

Changes in the treatments for disease and disability. 

Emergence of new or improved medical technologies and medical products such as 
vaccines, diagnostics, and therapeutics. 
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New Legislation and Regulations 



None. 



Key External Factors 

The rate at which scientific findings are adopted by the private sector and translated into 
commercial products and processes affects the transfer of research findings into practice. 
Moreover, scientific discovery and advancement are inherently unpredictable. 



Research addressed to controlling 
the spread of communicable 
diseases, eliminating the 
environmental causes of illnesses, 
and promoting health behaviors 
that forestall illness and premature 
death also is essential to HHS’s 
efforts to improve the health and quality of life of the nation’s citizens. Within the 
Department, the CDC is the focal point for population-based, public health research. This 
research is used by the Department and public health agencies throughout the nation to 
solve public health problems and improve the effectiveness of prevention programs. For 
example, research on the prevention of sexually transmitted diseases and childhood lead- 
poisoning, translated into public health practice, has significantly advanced the control of 
those diseases. 



Strategic Objective 6.3 
Improve Public Health Prevention 
Efforts through Population-Based 
Research 



Strategy for Achieving Objective 



The Department believes that continued investment in prevention research is essential to 

improvements in public health. Therefore, HHS will take these steps: 

■ sustain its priority of investment in population-based, public health prevention 
research; 

■ implement mechanisms to ensure that prevention research is responsive to 
emerging public health needs; and 

■ promote the collaboration between public health departments and academic 
institutions in the conduct of prevention research. 
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Measures of Success 



■ Number of new, innovative prevention strategies that are adopted into public 
health programs. 

■ Degree to which prevention objectives of Healthy People 2000 and 2010 are met. 



New Legislation and Regulations 



None. 



Key External Factors 

A number of external factors will strongly influence the accomplishment of this objective. 
They include such variables as changing public health needs, including the impact of new 
and re-emerging infectious diseases, and changing demographics, including the 
increasing elderly population and the growing incidence of chronic diseases. 

Increasingly, the prevention and treatment of diseases centers on behavioral changes, yet 
effecting permanent changes (such as reducing high-risk behaviors that can lead to 
disease and the adoption of health-enhancing behaviors) remains enormously difficult 
Developing successful research networks with the participation of state and local health 
agencies, health providers, and academic institutions is necessary to carry out population- 
based research. 



The nation’s health care system is 
undergoing a dramatic 
transformation as a result of rapid 
mergers and acquisitions among 
hospital systems, pharmaceutical 
firms, health plans, and managed 
care firms. Health care delivery 
systems are larger and more 
complex; cost pressures are forcing innovation in how and where care is provided; and 
the population is getting older. 

Health services research, which examines everyday practice in health care, is needed to 
assess the effect of current change and innovation on the cost, quality, and effectiveness 
of health care services. Many important questions need to be answered: Have the changes 



Strategic Objective 6.4 
Increase the Understanding of and 
Response to the Major Issues Related to 
the Quality, Financing, Cost, and Cost- 
Effectiveness of Health Care Services 
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made a difference in who is getting care, how much care is being provided, or what types 
of services offered? What is the cost and quality of the care being provided? Which 
services are most cost-effective? How can research findings on effective treatments 
move more quickly into the everyday practice of providers? New methods and products 
for assessing quality of service, patient satisfaction, efficiency, and outcomes are also 
needed. Health services research is important in providing the analytical foundations for 
making payment reforms, especially in federal programs such as Medicare. 



Strategy for Achieving Objective 

HHS will increase its understanding and response to the major issues in health services 

through these means: 

■ investing in health services research; 

■ relying increasingly on the model of investigator-initiated, peer-reviewed research 
to identify key research questions and select the highest-quality research proposals 
through processes that include workshops and forums in cross-cutting areas of 
research, regular internal and external reviews, and review of the effectiveness of 
support mechanisms; and 

■ creating research partnerships with states and private sector organizations. 

Measures of Success 



■ Availability of new tools and performance measures useful for assessing the 
quality of care provided in a range of health care settings, including measures of 
effectiveness of clinical care that recognize the role of patients in decisions 
affecting their treatment 

■ Availability of new, useful knowledge about the impact of changes in the 
organization and financing of health care. 

■ Availability of new, useful knowledge about the cost and cost-effectiveness of 
alternative treatments for common, high-cost conditions, especially conditions 
affecting the Medicare population. 
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Availability of evidence about the relevance of research results and about the 
implementation levels of research results by the public, practitioners, and the 
health policy community. 



New Legislation and Regulations 



None. 



— Key External Factors 

Collaboration with other federal agencies, as well as with states and the private sector, is 
essential to ensure that health services research is responsive to current and future 
challenges in the health care marketplace. 



One of the critical missions of the 
Department is to protect the public 
health. The Department performs 
this role in a number of ways, 
including the review and approval of 
new drugs and medical products 
developed by private sector research. 
The review and approval process, 
managed by the Food and Drug Administration (FDA), ensures the effectiveness and 
safety of the products being brought to the marketplace. 



Strategic Objective 6.5 
Accelerate Private-Sector Development 
of New Drugs, Biologic Therapies, and 
Medical Technology 



Recent public debate has questioned the length of time it takes for new health products to 
be developed and made available to the American public. The Department recognizes the 
tension between the need for a process adequate to protect health from dangerous or 
ineffective products and the benefits from moving products expeditiously to the 
marketplace. It nevertheless believes that the gains to health from accelerating research 
and development and marketing are compelling. The Department believes it is important 
to enhance the benefits to the public health by assisting the medical industry in 
accelerating the research and development process and in moving health products 
expeditiously to the marketplace. 
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Strategy for Achieving Objective 

The Department envisions a multipart strategy, through the FDA to assist the private 

sector in reducing the time required to bring new drugs and medical products to market. 

Specifically, FDA will take these steps: 

■ Concentrate on, and give priority to, those products that have the greatest potential 
for health benefits and that address serious or life-threatening illnesses. This step 
will ensure the widest public health benefit from expedited reviews. 

■ Speed up all reviews by transforming the regulatory review process for new drugs 
and biologic therapies into an electronic, paperless system. 

This process will reduce the time and effort drug and therapy sponsors and FDA 
staff spend in preparing and handling large volumes of paper. For example, the 
average new drug application has about 250 volumes of information and occupies 
about 46 linear feet of shelf space. Besides saving time and costs, electronic 
handling will enhance reviewer productivity by increasing the speed and 
convenience of analysis, including risk-assessment supporting review decisions 
and the review process itself. 

In the risk-assessment phase, automation will improve understanding of the way a 
new drug or biological product works, and of its benefits and potential role in 
health care. In the review stage, electronic data handling will eliminate pounds of 
paper and make it easier for a reviewer to access information within the agency, as 
well as to communicate with the sponsor. The Department expects a direct 
correlation between the degree of automation achieved and the time saved in the 
review process — and, by extension, time saved by the industry in development. 

■ Increase and improve communication with sponsors. These improvements will 
help speed up sponsors’ responses to FDA questions or concerns. By meeting with 
sponsors during the development of new products and the preparation of 
applications for review, FDA expects that applications will be complete and 
informative, thus reducing the down time now experienced while FDA reviews a 
completed application and prepares requests for clarification or additional 
information. 
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Harmonize U.S. regulatory standards with those of other industrialized nations. 

Product sponsors will be spared the time and expense of preparing applications in 
different formats and with varied data and analytic requirements. Mutual, 
recognition agreements will be used to promote a level playing field. 



Measures of Success 



■ Average time spent in overall product development processes. 

■ Percentage of complete priority biologies license applications and new drug 
applications reviewed within six months after submission date. 

■ Percentage of complete standard biologies license applications and new drug 
applications reviewed within twelve months after submission date. . 

■ Percentage of complete medical device pre-market approval applications reviewed 
within six months after submission date. 



New Legislation and Regulations 

Extension of the Prescription Drug User Fee Act is required to keep this objective on 
track. Many new regulatory standards will result from the harmonization process and will 
be integrated directly into the U.S. regulatory framework. Others will require new 
regulations. 



Key External Factors 

Outside factors that could affect the achievement of this objective include the health of 
the economy and its impact on the rate of industry’s investment in the development of 
new medical therapies. The rate at which new medical technologies emerge will also 
affect the speed of the development process, and so will the complexity of these 
breakthroughs. The rate at which medical therapies will be developed will also be 
affected by the nature, scope, and severity of emerging health risks. Harmonizing 
regulatory standards will depend on the international community’s coming to agreement 
on key issues. 
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Economic conditions, and the pace and direction of information and communication 
technology development, could have a bearing on FDA’s ability to automate the review 
process and also on the ultimate configuration of automation. Economic conditions, 
particularly in the private sector, could affect the pace with which industry systems could 
be successfully articulated with the agency’s systems. The direction taken by both 
information and communication technology over the next few years will influence the 
speed of automation and the configuration of automation. 



The presence of a cadre of talented 
individuals in medical science and 
health services research is essential 
to improving the health of the nation. 
A robust and diverse research 
workforce — in laboratory, patient- 
oriented, population-based, and 
systems research — is central to 
carrying out the Department’s 
mission and also to the vitality of present and future medical research enterprise and its 
associated industries. 



Strategic Objective 6.6 
Improve the Quality of Medical and 
Health Science Research by 
Strengthening the Base of Highly 
Qualified Scientific Investigators 



: Strategy for Achieving Objective 

The maintenance, enhancement, and renewal of the scientific talent base require the 
operation of strong research training and career development programs, a commitment to 
the recruitment and retention of under represented groups into science, and the provision 
of a nurturing and stable research environment. To accomplish these aims HHS will take 
these steps: 

■ invest in research training programs at institutions and fellowships for individuals 
through the National Research Service Award program; 

■ support mentored research experiences for clinical, laboratory, and health services 
research scientists through research career development fellowships; and 

H recruit minorities into medical research training and career development programs, 
starting with minority youth. 
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Measure of Success 

Numbers of highly qualified persons, including minorities, pursuing medical and 
health science research, in accord with the estimates of need by the National 
Research Council for medical research . 46 



— New Legislation and Regulations 

None. 



Key External Factors 

A number of external factors are important to the success of research training efforts. 
Demographic factors that shape the pool from which future scientists can be recruited are 
very important. In addition, it is difficult to attract talented young people if they perceive 
that the job market in research is unstable or less financially rewarding than are fields of 
comparable interest. Thus, the constraint on resources for research limits opportunities 
for training and career advancement and discourages new entrants into basic science 
careers. Finally, the universities that provide the training opportunities are undergoing 
enormous changes that will influence their ability to support and nurture young 
researchers. 



The expeditious communication of 
research results is a vital step in 
translating new knowledge into 
changes in medical practice and into 
technologies that improve human 
health and well-being. Effective 
communication of research findings 
drives scientific innovation, fosters 
new discoveries, and ensures that 

public investment in research yields 
new methods of prevention, diagnosis, and treatment. 



Strategic Objective 6.7 
Ensure That Research Results Are 
Effectively Communicated to the 
Public, Practitioners, and the 
Scientific Community 
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jtrategy for Achieving Objective 
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■ Improvements in dissemination of research findings and emergency or critical 
information learned from medical and health services research. 

New Legislation and Regulations 



None. 



Key External Factors 

The pace of advances in information technology is a key factor in improvements in 
communication. The capacity of national Web resources will limit public access to and 
use of HHS-sponsored resources. The success of efforts to connect schools, libraries, and 
other public facilities to the Web will also set the pace for public access to these 
information sources. 
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U.S. Department 
of Education 

Strategic Plan, 1998-2002 

September 30, 1997 



The Department of Education’s Mission 

To ensure equal access to education and to promote educational 
excellence throughout the nation. 
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Objective 1.4: A talented and dedicated teacher is in every classroom in America. 



Performance Indicators 

25. The percentage of teachers and principals across the nation who are rated by supervisors, 
parents, and peers as very effective will increase annually. 

26. Throughout the nation the percentage of secondary school teachers who have at least a minor 
in the subject they teach will increase annually. 

27. The percentage of qualified new teachers who leave the profession within the first 3 years will 
continuously decrease. 

28. The number of nationally board certified teachers will increase to reach 105,000 by 2006. 

29. By 2002, 75% of states will align initial teacher certification standards with high content and 
student performance standards. 

A talented, dedicated, and well-prepared teaching force is one of the most important ingredients for 
education reform. Research indicates that teachers’ knowledge and skill make a crucial difference in 
what students learn. Research also demonstrates the value of intensive and sustained high-quality 
professional development when developing skills in new models of teaching and learning. The current 
teaching force needs high-quality professional development if all teachers are to be able to teach a 
diverse student population to challenging standards. Further, new teachers must be well prepared to help 
diverse learners master challenging content and performance standards. It is estimated that about one- 
fourth of newly hired teachers lack the qualifications for their jobs. There is also high turnover in 
beginning teachers — 22% of beginning teachers drop out of the teaching profession within the first three 
years. Key reasons include lack of support and typical “sink or swim” approaches to induction. 
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Core Strategies 

• Improving the quality and retention of new teachers. 

- Support programs to recruit talented people from all backgrounds into teaching. 

- Improve the quality of teacher education for new teachers. 

- Encourage and support special efforts to retain new teachers. 

- Support career ladders that will enable bilingual paraprofessionals to become certified teachers. 

• Financial support and interagency coordination. Provide funds to states and schools through the 
Eisenhower and Individuals with Disabilities Education Act (IDEA) professional development 
programs, as well as other programs in which professional development is an allowable activity (e.g.. 
Title I, Vocational Education, Bilingual Education, and Technology Literacy Challenge Grants). 
Coordinate with the National Science Foundation to implement strategies to improve the skills of 
teachers through the professional development programs of both agencies. 

• Licensing standards. Through the Partnership for Excellence and Accountability in Teaching, 
support states’ efforts to align licensing and certification requirements with challenging content 
standards and performance-based assessments. 

• Teacher recognition and accountability. Support the National Board for Professional Teaching 
Standards and programs that reward good teachers and address the problems of incompetent ones. 

• Workplace knowledge. Promote public-private sector partnerships, such as industry-based 
internships, that increase teachers’ knowledge and skills of the changing workplace. 

• Research, development, evaluation, and dissemination. 

- The National Evaluation of the Eisenhower Professional Development Program will evaluate the 
program’s alignment with high state standards and principles of duration and intensity. 

- Provide educators and policymakers with valid, research-based strategies for improving quality. 

- Increase support for quality teaching and professional development among government, business, 
community leaders, and the general public through outreach and dissemination of information. 

- Establish the Partnership for Excellence and Accountability in Teaching to work with stakeholders 
and carry out applied research. 

- Establish a Teacher Policy Center to conduct research on policies related to teaching. 

Monitoring trends. Issue a biennial national report card on teacher quality starting in 1998. 
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Objective 1.6: Greater public school choice will be available to students and 
families. 

Performance Indicators 

35. By 2002, 25% of all public school students in grades 3-12 will attend a school that they 
or their parents have chosen. 

36. By 2001 a minimum of 40 states will have charter school legislation. 

37. By 2002 there will be 3,000 charter schools in operation around the nation. 

38. School districts will increasingly make choice available to their students through magnet 
schools, charter schools, and open enrollment policies. 

" Research on public schools that provide choice suggests that the sense of ownership by school staff, 
students, and parents helps to galvanize effort towards common goals. Information on the educational 
effects of choice programs is limited; most charter schools are just getting started. Further work needs to 
be done on documenting the implementation and quality of public schools of choice and sharing the most 
promising strategies with the field. 

The Department of Education is encouraging expansion of choice within the public school system with 
alternatives such as charter schools, magnet schools, and systemwide strategies that make every public 
school a school of choice, thereby enabling all students and their parents to choose their school. 

■ Charter schools are intended to give teachers, parents, and other members of local communities the 
flexibility to experiment with innovative methods of achieving educational excellence. At the same 
time, they should help all students have access to quality schooling. Because they are new schools, 
charters require start-up funds and support that the Department helps to provide through its Charter 
School program. 

■ For several decades, magnet schools have provided the most widespread opportunity for families 
to exercise choice. The Department’s Magnet School program provides support for magnet schools 
that are intended to achieve desegregation goals, particularly in our largest cities. 

Core Strategies 

• Engage the public. Expand support by the public and policymakers for the development of high- 
quality charter and magnet schools. 

• Financial support and technical assistance. 

- Through the Charter Schools Grants program, help states and schools effectively plan and 
implement charter schools that have flexibility from state and district rules, are open to all 
students, and are held accountable for improving student achievement. 

- Continue to support implementation of magnet schools through grants to school districts under the 
Magnet Schools program, providing opportunities and choice for students and promoting 
desegregation within high-quality education settings. 

• Research and development. Support research on public school choice, including evaluations of the 
effectiveness of charter schools and magnet schools, and promote the development of models and 
materials to help parents, teachers, and communities to design effective school choice programs. 

• Outreach. 

- Disseminate information on strategies for expanding high-quality school choice programs that 
improve student achievement and share lessons learned from research about school choice. 

- Increase awareness and support for effective public school choice programs among government, 
business leaders, and community leaders, and the general public through outreach and 
dissemination of information. 
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Objective 1.7: Schools use advanced technology for all students and teachers to 
improve education. 

Performance Indicators 

39. Students who have access to high-quality educational technology will show improved 
achievement in core academic subjects and improved technological literacy. 

40. The ratio of students per modem multimedia computer will improve to 5: 1 by 200 1. 

41. The percentage of public school instructional rooms connected to the Information 
Superhighway will increase from 14% in 1996 to 25% in 1998, and higher percentages 
thereafter. 

42. At least 50% of teachers will integrate high-quality educational technology, high-quality 
software, and the Information Superhighway into their school curricula, by 200 1. 

43. Students in high poverty schools and students with disabilities will have access to advanced 
technology (including assistive technology for students with disabilities) that is comparable to 
that in other schools by 2001. 

44. At least 60% of teachers, school administrators, and school librarians will have been trained 
on use of computers and the Internet to help students learn, by 2001. 

Hundreds of studies have found that, when properly used, technology improves many aspects of 
education, including student learning, teacher professional development, classroom management, and 
school administration. As an instructional tool, technology helps students master basic skills, solve 
complex real-life problems that require advanced skills, and prepare for the world of work. 

Few schools have adequate numbers of modem computers or access to the Internet, and relatively few 
teachers are prepared to use technology effectively. Further, access to computers and other technologies 
is not enough; integration of technology into the curriculum is also needed. We must create an 
infrastructure that will enable all students to leave school with the technology skills needed for work and 
further education. Finally, we must encourage development of software and universal design interfaces 
that make advanced technology fully accessible to students with disabilities. 

Core Strategies 

• Technology connections, especially for high-poverty schools. 

- Use the Federal Communications Commission’s Universal Service Fund discounts and “NetDays” 
to wire schools for using educational technology and to connect them to the Internet. 

- Encourage use of technology connections, such as voice mail, faxes, and e-mail, to stimulate 
communication between families, communities, teachers, and schools. 

• Access to modern computers and other technology. 

- Encourage local, state, federal and private sector partnerships to provide access to modem 
computers for all teachers and students, including those in high-poverty schools. 

- Provide financial support through the Technology Literacy Challenge Fund and other programs to 
states and districts to plan, purchase, and use modem computers and other educational technology. 

- Support development, dissemination, and use of assistive technology that enables students with 
disabilities to participate fully in education programs. Key programs include research by the 
National Institute on Disability and Rehabilitation Research and support from the Assistive 
Technology program and IDEA. 

• Effective software. Using state and local standards as guides and building on research and 
development of effective practices, including those developed with support of the Technology 
Literacy Challenge Grants, work with the private sector to develop effective and engaging software 
and on-line learning resources as an integral part of school curriculum. 

• Program coordination. Through our technology initiative, coordinate Department technology 
programs (Technology Challenge programs, regional consortia. Star Schools, IDEA technology and 
media services, assistive technology, Ready-to-Leam Television, and telecommunications math 
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programs); other programs that can support technology, such as Title I and IDEA, and programs and 
services in other federal agencies such as the National Science Foundation. 

• Professional development. Building on new teaching standards, support teacher training through 
federal programs such as Eisenhower Professional Development, Technological Literacy Challenge 
Fund, Star Schools, Bilingual Education, Vocational Education, and Title I programs, and by working 
with the National Science Foundation. In partnership with states, local districts, and the private sector, 
create new incentives and approaches and provide technical assistance that will help teachers use 
technology more effectively. 
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Objectives, Indicators, and Strategies 



Objective 2.1: All children enter school ready to learn. 

Performance Indicators 

1. Kindergarten and first grade teachers will increasingly report that their students enter school 
ready to learn reading and math. 

2. The disparity in preschool participation rates between children from high-income families and 
children from low-income families will decline year by year. 

3. The percentage of children from birth to five years old whose parents read to them or tell them 
stories regularly will continually increase. 

Recent research has highlighted the importance of the earliest years of life for children’s later success. 
Children’s early learning experiences, or lack of them, have consequences that extend into the long-term. 
Research on early brain development reveals that if some learning experiences are not introduced to 
children at an early age, the children will find learning more difficult later. Furthermore, children who 
enter school ready to learn are more likely to achieve to high standards than children who are 
inadequately prepared. High-quality preschool and child care are integral in preparing children 
adequately for school. 

Core Strategies 

• Interagency coordination, including services integration. Support children at risk of early school 
failure by coordinating with the Department of Health and Human Service’s (HHS) Head Start 
program, HHS’ and Department of Agriculture’s nutrition support programs, and other federal 
programs and services for young children to ensure that their needs are met and to reduce the burden 
on families and schools of working with multiple providers. 

• Financial support for children who are educationally disadvantaged or have disabilities. Provide 
resources to states and local school districts under Title I for preschool programs and Even Start, and 
to states and local providers under the Individuals with Disabilities Education Act (IDEA) for 
programs aimed at infants and toddlers and preschool children with disabilities or at risk of 
developing disabling conditions. 

• Research, development, and technical assistance. 

- Identify, evaluate, and encourage the use of programs for young children that make use of the 
latest research on early brain development, early intervention, and high-quality nurturing. 

- Develop, field test, and evaluate models of effective practice through such programs as Even Start 
that can be shared with local Head Start, Title I preschool, and IDEA preschool projects and with 
states, local districts, and community-based organizations. 

- Work with experts to develop an agreed-upon definition of school readiness and to establish a core 
set o'f standards that Even Start, Title I preschool, and IDEA programs will use with preschoolers. 

• Development and dissemination of easy-to-use kits for learning at home. Support family practices 
that encourage early learning by developing and disseminating educational materials for parents and 
their young children, such as the Ready*Set*Read Early Childhood Kit. 

• Development of readiness indicators. Develop indicators of young children’s knowledge and school 
readiness by working with HHS and other organizations, incorporating measures from the Early 
Childhood Longitudinal Study and other studies of children’s school readiness. 
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Objective 2.2: Every child reads independently by the end of the third grade. 

Performance Indicators 

4. Increasing percentages of fourth-graders will meet basic, proficient, and advanced levels in 
reading on the National Assessment of Educational Progress (NAEP). 

5. At least 25% of students will participate in the national reading test by spring 1999; 
increasing percentages thereafter will participate. 

6. By 2001 the America Reads Challenge corps will prepare tutors for 3 million children, 
including at least 100,000 college work-study tutors annually. (Legislation needed.) 

7. Increasing percentages of teachers of kindergarten through third grade will complete 
intensive professional development to enable them to skillfully teach reading. 

In 1994, 40% of fourth-graders failed to attain the basic level of reading on the National Assessment of 
Educational Progress and 70% did not attain the proficient level. Although reading problems are 
particularly severe for disadvantaged students, students with reading difficulties represent a cross-section 
of American children. As more and more jobs require better reading skills, many students will have to 
improve their reading skills. 

The Department's existing programs make a vital contribution to the reading success of young children. 
Title I of the Elementary and Secondary Education Act provides reading services to millions of children 
each year. The Individuals with Disabilities Education Act and bilingual education funds under Title VII 
also support reading services for children. Although teachers and schools have the critical responsibility 
for literacy, studies find that sustained, individual attention and tutoring after school and over the 
summer can raise reading levels when combined with parental involvement and quality school 
instruction. 

Core Strategies 

• Legislation. Work with Congress to pass the America Reads Challenge legislation. 

• Financial support for children with special needs. Provide in-class reading instruction with 
upgraded standards and curriculum — especially for children in kindergarten through third grade. Key 
programs that support reading instruction include Tide I Grants for Disadvantaged Children, Bilingual 
Education, the Individuals with Disabilities Education Act, and Even Start. 

• Voluntary national test. 

- Support the development and effective use of a national, voluntary test in reading so that parents, 
teachers, and communities have a benchmark for children’s progress. 

- Provide accommodations for students with disabilities and limited English proficiency in taking 
the national test, including providing reports for parents in English and several other languages. 

• Public information. Provide information via the world wide web and other means to bring about an 
understanding of what it means to read independently and share strategies that teachers, parents, and 
others can use to help students achieve this goal. 

• Community partnerships. Encourage community partnerships that sponsor reading tutors (the 
America Reads Challenge, Read*Write*Now, Parents as First Teachers, and college work-study). 

• Research and development. 

- Support state-of-the-art research — including a reading center — to test, validate, disseminate, and 
encourage the use of effective approaches to reading instruction and tutoring, especially for 
students experiencing difficulty with reading. 

- Coordinate with reading research conducted for children with learning disabilities by the National 
Institute for Child Health and Development (NICHD). 

• Evaluation and performance measurement. Through evaluation studies and support to improve 
state and local performance data systems, provide useful information on how states and communities 
are doing in improving children’s reading. 
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Objective 2.3: Every eighth-grader masters challenging mathematics, including 
the foundations of algebra and geometry. 

Performance Indicators 

8. More eighth-graders reach the basic level or higher levels of proficiency in math on the 
National Assessment of Educational Progress; on international assessments, at least 60% will 
score at the international median by 2002, and at least 15% will be in the top 10% by 2002. 

9. At least 25%, of students will participate in the national math test by spring 1999; increasing 
percentages thereafter will participate. 

10. Each year, more new teachers will enter the workforce with adequate preparation to teach 
challenging mathematics to students in kindergarten through eighth grade. 

11. Each year, more teachers in grades 5-8 will complete intensive professional development to 
enable them to teach challenging mathematics. 

12. Each year, increasing numbers of schools will have access to and use information on best 
practices for math instruction. 



Mathematics is a basic skill— the gateway to learning many more advanced skills, the language of 
technology and science, a tool for analysis and problem solving, and a prerequisite for success in a wide 
variety of careers. Leading employers emphasize the need for U.S. students to excel in quantitative and 
problem-solving skills in order to succeed in the workplace. Math, like reading, has a key academic 
turning point; for math this occurs around eighth grade. Eighth-graders are often put on different tracks 
that they follow through high school and even beyond; math often determines what that track will be. 

/ 

Notwithstanding math’s importance, U.S. students fail to achieve to the high standards needed for math 
success. The recent Third International Mathematics and Science Study (TIMSS) showed that although 
U.S. fourth-graders perform above the international average in math, our eighth-graders scored below the 
international average. The study also showed that we do not expect eighth-graders to master material as 
challenging as the material that students in high-performing nations master by that grade. 

Core Strategies 

• Voluntary national test. 

- Support the development of a national, voluntary test in math so that parents and communities 
have a benchmark for their children’s progress. 

- Use the test as a means of encouraging schools, districts, states, business, and communities to 
improve math curricula, instruction, teacher training, and professional development. 

- Provide accommodations for students with disabilities and limited English proficiency in taking 
the test, including providing reports for parents in English and several other languages. 

• Professional development programs. Strengthen the Department’s existing programs that support 
teacher preparation and upgrading teacher skills for math instruction— especially for teachers of 
fourth through eighth grades— such as Title I of the Elementary and Secondary Education Act, the 
Eisenhower Professional Development program, and the Individuals with Disabilities Act 
Professional Development. 

• Challenging standards. Promote upgraded standards and curriculum for math instruction through 
Goals 2000, the Eisenhower Professional Development program, and Title I of the Elementary and 
Secondary Education Act, and by working with the National Science Foundation. 

• Public information. Increase public understanding and support for mastering challenging 
mathematics by the end of eighth grade through partnerships with key education, mathematics, and 
professional organizations; further collaborative activity with the National Science Foundation; and 
providing concrete information about what students should be able to do in mathematics. 

• Research, development, and dissemination. Based on state-of-the-art research, develop high quality 
materials on effective practices and tools for improving math curriculum, professional development, 
software, instruction, and family and community support; widely disseminate these materials; and 

O mote the use of these materials by states, schools, teachers, and families. 



Objective 2.4: Special populations receive appropriate services and assessments 
consistent with high standards. 

o 



Performance Indicators 

13. States will implement appropriate procedures for assessing and reporting progress towards 
achieving to high standards by students who have disabilities, are limited English proficient, 
or are children of migrant workers, by 2001. 

14. The number of schools using comprehensive, research-based approaches to improve 
curriculum, instruction, and support services for at-risk students will increase annually. 

15. Increasing percentages of administrators and educators working with at-risk children will 
have access to and use high-quality information and technical assistance on effective practices 
provided by Department-sponsored technical assistance and research centers as well as 
through professional associations and publications. 

16. Increasing percentages of teachers will be equipped with strategies to enable students with 
limited English proficiency or disabilities to meet challenging standards. 

17. Federal technical assistance and other support to states will result in annual increases in the 
number of states and local school districts with the capacity to disaggregate and report out 
assessment data aligned with standards for at-risk students. 

At-risk children need the same high quality schooling that is our goal for all students plus extra supports 
to help them succeed. These children may include students with limited-English proficiency or 
disabilities, migrant students, students in high-poverty schools, and others who are the focus of federal 
programs. Federal support is critical to ensuring that these students are not left behind in the drive for 
higher standards. Working to enable at-risk children to reach the high standards expected of all students 
must figure prominently in reform efforts. Whole-school approaches or targeted interventions must be 
based on the best research and promising practices from the field. Assessment of our nation’s progress 
must be measured in terms of not only how well states, districts, and schools perform overall, but also in 
terms of how at-risk students fare. 

Core Strategies 

• Challenging standards in federal programs. Work with states and districts to ensure that the 
standards set for students served by federal programs are the same challenging standards set for all 
children through providing technical assistance, guidance, and models of effective implementation of 
challenging standards. 

• Assessment with, accommodations. Promote the development of assessments aligned with high 
standards that make appropriate accommodations for children with disabilities and limited English 
proficiency. 

• Financial support. Provide significant resources to states, local school districts, and other education 
providers to improve achievement for children with special needs and assist states in providing 
education that meets civil rights requirements for free and appropriate education. The Department 
funds major programs aimed at disadvantaged children or children with disabilities, including: 

- Title I of the Elementary and — Programs for homeless children and youth 

Secondary Education Act (education - Indian education 

for disadvantaged children) - Bilingual education 

- Migrant education programs - Individuals with Disabilities Education Act 

- Title I program for neglected and (IDEA) program 

delinquent children 

• Research, development, dissemination and technical assistance on promising practices. Support 
and share research on the most promising practices through the research institutes and R&D centers of 



Outcome indicators for children and youth in special populations are included 
throughout Goals 1 and 2 and especially in the key outcome indicators for 
elementary and secondary education on page 15. 
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the Department’s Office of Education Research and Improvement (OERI), Office of Special 
Education and Rehabilitative Services (OSERS), the Office of Bilingual Education and Minority 

L ^u U ?i?f A , ffairS (° BEMLA )> ^ die Department of Health and Human Services’ National Institute 
of Chdd Hcahh and Human Development (NICHD) to focus on strategies for teaching and assessing 
children with special needs. In particular, provide technical assistance and disseminate information on 
inc u ing children with disabilities in the general cumculum in the least restrictive environment. 

• Professional development. Support professional development that equips teachers with strategies to 
enable students with limited English proficiency or disabilities to meet challenging standards. Key 
programs include professional development programs sponsored under the Individual with 
Disabilities Act and Bilingual Education Act, as well as that provided under Title I. 

• Evaluation and continuous improvement. 

Conduct evaluations of federally supported programs to determine the extent to which new 

program provisions support standards-based reforms and continuous improvement to help students 
meet challenging academic standards. 

- Use evaluations to inform continuous improvement of programs. 
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Objective 4.3: An up-to-date knowledge base is available from education research 
to support education reform and equity. 

Performance Indicators 

10 Pggr reviews will increasingly show that education research and statistics supported by 
the Department are of high quality, are focused on critical education reform issues, and 
contribute significantly to educational improvement. 

11. Education research will increasingly meet the needs of our partners (e.g., states, 
schools, institutions of higher education, national associations) and our customers 
(teachers, parents, students, business) for reliable information on how to make schools 
more effective, as measured by biennial customer surveys. 

1 2. In major and selected other programs, increasing percentages of grantees will 
demonstrate that their programs and services are based on sound research results . 

13. Dissemination of research and assessment findings will increasingly reach key 
customers and result in educational improvement. 

Investing in education research and evaluation contributes to our understanding of and efforts to improve 
education. Because of its potential to influence the well-being of the nation’s youth, education research 
must meet the highest professional standards of scientific inquiry so that results are trustworthy. The 
Department, in collaboration with the National Educational Research Policy and Priority Board, is 
developing standards to assure that supported activities are of the highest professional excellence. To 
ensure its relevance and application, research must remain firmly rooted in the everyday experience of 
students and teachers and the reality of schools. The Department also supports a variety of national 
dissemination activities that make available to educators, parents, and policymakers as well as ED 
program staff— the best research-based information on educational practice. 

Core Strategies 

• Statistics. Collect and effectively disseminate statistics on critical education issues used to inform the 
national research agenda and provide information for policy-making and program improvement. 

• National vision and priorities for research. 

- Develop a comprehensive vision of the nation’s needs for knowledge about education, and set clear 
priorities for education research to meet those needs. 

- Coordinate research, development, and evaluation activities across the Department and with other 
federal agencies, such as the National Science Foundation and HHS institutes. 

• Financial support for R&D. Support research on education reform and improvement through such 
programs as the national education research institutes and centers, regional educational laboratories. 
National Educational Research Policy and Priorities Board, National Institute for Disability and 
Rehabilitation Research, IDEA Research to Practice program. National Center for Research in 
Vocational Education, and the International Education and Foreign Language Studies program. 

• Research quality. _ 

- Ensure that Department-supported research and development meet the professional standards of 

the scientific community and are applied systematically and with rigor. 

- Develop and utilize knowledge about education systems and practices in other nations to stimulate 
educational improvement in the United States. 

• Research dissemination and use. 

- Develop and implement a comprehensive dissemination system of effective practices that increases 
the education community’s access to and use of research-based products and services. 

- Ensure that teachers, parents, and principals can obtain help in solving their school-related 

problems. , 

- Review and give feedback on the extent to which the Department’s grantees propose programs and 

services that are based on sound research results. 

- Ensure that research and program evaluation findings are given to program offices to improve 
program design and implementation. 




DRAFT 



A REVIEW OF THE FEDERAL ROLE 
AND THE DEPARTMENT OF 
EDUCATION STRUCTURE 



Submitted to: 

National Educational Research Policy and Priorities Board 
US Department of Education 
80 F Street, NW, Suite 100 
Washington, DC 20208 

Through: 

American Institutes for Research 
Washington, DC 

Submitted by: 

Mathtech, Inc. 

202 Carnegie Center, Suite 111 
Princeton, NJ 08540 

March 10, 1998 




150 



A REVIEW OF THE FEDERAL ROLE AND THE 
DEPARTMENT OF EDUCATION STRUCTURE 



This fourth and last of four background papers for the Research, Development and 
Dissemination System (RD&D system) Committee of the National Educational Research Policy 
and Priorities Board (NERPPB) provides a summary review of the federal role in research (and 
specifically educational research) and the structure (organization and processes) for fulfilling the 
role in the U.S. Department of Education (ED). 

This paper is based in part on: the 1992 review by a panel of the National Research 
Council (NRC), Research and Education Reform, Roles for the Office of Educational Research 
and Improvement; the 1991 review by a committee of the National Academy of Education 
(NAE), Research and the Renewal of Education; the current law; selected commentary; and a 
recent review of the Department’s knowledge building processes and its Planning and Evaluation 
Service (Morrill & Weiss, Talent, Tensions and Transition, An Organizational Analysis of the 
Planning and Evaluation Service, February 1997). It is also grounded in its author’s experience 
and analysis of RD&D activities within and outside of the federal government. 

The review begins with consideration of the appropriate federal role, turns next to the 
effectiveness of the ED structure and concludes with implications of the analysis for the Board’s 
Congressional report. 

I. The Federal Role 

To set a proper context for examination of the federal role in educational RD&D, we 
begin with a summary of the theory and concepts determining appropriate federal activity which 
have been applied in American governance with varying degrees of rigor over time, and which 
have gradually assumed a more expansive view. 

The historic baseline has been to limit the intervention of all governments to undertaking 
only those activities whose purposes are unattainable in the desired amount or quality through 
private action and where the public benefits equal or exceed the public costs of production. 
Through this lens, education - particularly elementary and secondary education - has long been 
regarded in the economic and political science literature as a public good. The benefits to the 
society of an educated citizenry for economic, social and aesthetic reasons have long had sub- 
stantial acceptance among scholars and the general public alike. Notwithstanding current debate 
about the broader introduction of private provision of K-12 education with public money (e.g., 
vouchers), the consensus about the need for public education financing remains intact. 

The second threshold question once public action is judged appropriate, is the determina- 
tion of the appropriate level of public intervention - local, state, federal, or some combination 
thereof. Traditional American political doctrine would remand the public intervention to the 
lowest level where the function can be effectively performed. Economic analysis would examine 
whether the benefits realized are local, regional or national in determining where the costs should 
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fall. On these grounds, there is a basis for funding shared benefits at all three levels, with the 
higher the level of educational attainment justifying a larger contribution from higher govern- 
mental levels, since local government usually doesn’t capture the full benefits of higher educa- 
tion of its resident children. This conceptual framework has, in general, been followed in support 
of education, with the state and local governments carrying most of the K-12 financial load, and 
the federal and state levels carrying most of the public postsecondary burden, beyond costs 
carried by individuals. 

In addition to the criteria outlined above, there is another set of criteria in assessing 
federal involvement applicable to the kind of activity being undertaken. This additional criterion 
also has its roots in economic cost and benefit notions as well as political science concepts. 
Activities can be seen in a continuum from research and information at one end, through to 
directly provided federal services or income transfers at the other end. In this continuum, 
research and information are the most easily justified as a federal role and the direct interventions 
the hardest to justify. Justification of the research and information roles involve analysis of 
whether research and information needed for service improvement or consumer choices are 
available from either private or other public parties. And in the absence of other sources, federal 
research and information activity is considered warranted. 

This conceptual framework has had broad application across the federal government, 
influencing involvement and levels of expenditures from relatively high proportionate levels in 
defense, space, substantial but not all dimensions of health, down to activities where few or no 
federal investments are made because of commercial or other alternative sponsorship of 
appropriate R&D. Against this criterion, educational RD&D should be highly ranked in 
eligibility for federal support. 

Given the public nature of the enterprise - particularly K-12 - and the difficulty of 
capturing all but the most applied research benefits in particular institutions, the case for RD&D 
sponsorship from public and philanthropic sources is strong. Further, locating the bulk of public 
sponsorship at the federal and state levels is warranted given budget constraints, skill limitations, 
efficiency considerations and the inability to capture enough of the benefits of the research at the 
school district level. Also unlike the drug companies, educational vendors have limited capacity 
or economic base for large R&D investments. 

This is not to say that state and local government, educational vendors, and foundations 
do not have an important interest and stake in educational RD&D. They do, and their role in 
each case, including financial contributions, should be stronger. It is to suggest, however, that 
the existing structure beyond the federal government does not encourage large step-wide changes 
in sponsorship of educational RD&D without very large changes in the underlying financing 
system and incentives, which seem unlikely in the critical next 5-10 years. Only in the case of 
philanthropic giving does significant adjustment seem plausible, and probably only if there is an 
accompanying increase in federal funding. 

The foregoing suggests that there is a conceptual base and justification for a much larger 
federal role and federal financial support level in educational RD&D than now exists. 
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Intellectually competent reviews over the past decade (NAE, NRC and the President’s 
Committee of Advisors on Science and Technology (PCAST)) have called for such significant 
increases. In earlier papers, we have discussed some of the reasons for lack of response; here, we 
need to examine the non-response in terms of the federal role. 

On those terms, the federal role in educational R&D has primarily run into difficulty over 
the presumed linkage between sponsorship of research and information on the one hand and the 
direction of operational activities on the other, with particular focus on curriculum. Americans 
have long held strong views against federal operational control of K-12 schooling, in sharp 
contrast with a number of European democracies and other countries. The presumption holds 
that federal sponsorship of research (or testing for that matter) will lead inevitably to operational 
control inclusive of curriculum, a possibility so problematic as to warrant foregoing the research 
altogether. Adults know “what worked” or didn’t for them, worry about school competition in 
the values of their children, and are perhaps unimpressed with the science base of education 
research. 

This argument is further muddied by concerns about “politicization” of the research, a 
concern that the research community as well as the political (Congressional) community shares. 
While certainly not the only barrier to more vigorous federal educational R&D, it is generally 
recognized as an important one. 

It is interesting to see where and how this kind of concern exhibits itself in federal R&D 
and where it does not, and speculate about the causes. 

• This objection does not occur in many other highly decentralized activities in which 
operational control is firmly vested in the operational level. Health and agriculture 
are both highly decentralized activities in which federal research plays a strong, if 
not dominant role. It is to be noted that the health system also deals with children; 
and in other social services, federal R&D is not regarded as an anathema. Perhaps 
this is due to the fact that the health system primarily fixed bodies, not minds. 

• Even in education, there is a differential set of standards applied. With respect to 
special education and vocational rehabilitation, very substantial federal sponsorship 
exists and in the case of NICHD, is warmly supported even by the House 
Committee on Education and the Workplace. 

• Work on math and science education including curriculum by NSF receives support 
by the Congress, as does educational research in other mission agencies. 

These examples suggest that something more or different from the traditional concern about 
federal involvement in decentralized operational activities are at work, even in the educational 
R&D area. Some of the candidate factors which cause the hostility to federal educational R&D 
when sponsored by OERI might include: 
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• A mind set among the lay public and political community that there is nothing 
fundamental or very useful to be learned from educational research that can help 
educational performance. In this view, it’s all a matter of minor applied tinkering 
and fads and not worth very much; and/or 

• The educational problem in the country except at the urban and rural fringes is in 
good shape, and the fringes are not very important or remedial with research; and/or 

• The quality of educational R&D management and performance is so poor as to be 
worth relatively little support; and/or 

• The ED oversight of federal R&D is so couched in partisan rather the substantive 
research terms that it is not to be trusted with resources since they will merely 
initiate the next fad, a fire that the research community itself often fans. 

These alternative sources of explanation for less support for federal educational R&D 
than might otherwise be the case are all unpleasant and at least partially untrue. They, 
nonetheless, constitute a formidable barrier which needs to be confronted and a strategy 
developed, if a change in the vigor, direction and size in federal educational research is to be 
achieved. That such a change is possible from a position of substantial criticism can be seen in 
the progress in NCES over the past decade. 

While approaches to the first three alternative and speculative causes of the suboptimum 
circumstance will show up in the last section, it is appropriate to deal with the “politics” issue 
here as an analytical matter. Initially touched upon in the second background paper (The Needs, 
Demand For and Supply of Educational Research, Development and Dissemination, February 
27, 1998, pp. 9-10), the political concerns often are directed at higher levels of the Department 
and executive branch in the form of alleged improper intrusions into matters of agenda, technical 
choices, and other details of scientific merit. Some, but not all, intrusions are improper; the 
Congress is at least as large a transgressor as the executive branch in micro-management; and the 
efforts to “wall-off’ elected and appointed sponsors from substantive policy guidance is a losing 
strategy not practiced by effective research managers. 

• Neither senior executive branch officials nor the Congress should have a direct role 
in project selection, evaluations, or other issues of technical merit. 

• Both senior executive branch officials and the Congress should abandon or curb 
their predilection for detailed allocation formulas and authorization of large-scale 
demonstration of untested and unresearched concepts and innovations. 

• Both senior executive branch officials and the Congress, however, should 
appropriately be involved in the policy issues of setting research priorities, broad 
policies and learning agendas in response to proposals of senior research managers. 
These are appropriate roles for sponsors, and effective research managers elsewhere 
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These are appropriate roles for sponsors, and effective research managers elsewhere make 

positive use of their participation. 

II. The Departmental Structure 

Reorganization or “moving the boxes around” is among the most popular activities in 
Washington, and it has been practiced on OERI - the central entity for educational RD&D - with 
regularity and excruciating detail by all interested parties, most particularly the Congress. 
Unfortunately, experience across the federal bureaucracy suggests that reorganization alone is 
among the least effective ways to produce the desired change for which it is undertaken. Without 
accompanying clarification of mission, improvements in approach and process, adjustments in 
skills, staffing and resources, and wise strategies, reorganization is a hollow shell. 

The present OERI organizational structure owes much to the NRC and NAE reviews of 
the early 1990s even after those ideas had been pushed through the compromising process of 
federal legislative enactment. The NRC organizational prescription was particularly detailed and 
explicit; and much of it survived into law, without, it should be noted, much of what it had to say 
about staffing and financial resources. These structured reviews were complemented by similar 
prescriptions from informed scholars (see Hawley, 1990; Wise, 1990). It is striking to read the 
remarks of representatives of this same group of informed observers in the immediate aftermath 
of the legislative enactment in 1 994 directed to a common set of questions about next steps 
beyond the legislation (Sroufe, et al., 1995). Those comments were focused on expansive 
dreams for educational RD&D at the federal level, instruction to stakeholders about their desired 
behavior, and very little discussion of how the reorganization would help achieve those goals or 
how the absence of other prior prescriptions were likely to diminish the chances of achievement. 

Before discussing the structural issues applicable to OERI in particular, it is appropriate 
to begin with the structure and processes of the Department as a whole. The organizational 
structure is thoroughly conventional and similar to many other domestic agencies. The Secretary 
is equipped with the normal set of alter egos - a Deputy and an Under Secretary - and staff 
offices, among them an analytic shop (the Planning and Evaluation Service). There are an array 
of program assistant secretaries with particular functional responsibilities and programs to run. 
And there is an assistant secretary for R&D, and a separate and especially insulated information 
gatherer (NCES). 

There are two features that distinguish the ED structure from many other agencies. The 
first is the insulation that several programs have - with statutory backing - to protect them from 
the Secretary and Secretarial action with respect to direction and resources. These include 
special education, vocational rehabilitation and research. The discussion will return later to 
research; but with respect to the other two, the underlying intent appears to be program 
protection and autonomy. Both have substantial R&D programs beyond the normal amount of 
knowledge building capacity, demonstration efforts, and dissemination activities (technical 
assistance) which program assistant secretaries should and usually do have. (See Morrill & 
Weiss, 1997 for further discussion). The insulation goes so far as to preclude evaluation activity 
by the Planning and Evaluation Service in special education. 
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The second anomaly has been, until recently, the absence of any planning process to pull 
the knowledge-building components of the Department together into a cohesive and coordinated 
strategy to develop the knowledge needed by the field and the Department for sound policy and 
operational improvement. In the absence of such a process, the system will focus on who is 
important rather than what is important. Resources will tend to reinforce the status quo rather 
than move in concerted directions. As noted in the review of these processes, OERI and NCES 
were treated more like foreign principalities for whom ambassadors were the mode of 
communication rather than dialogue among members of the same department. The attitudes 
imbedded in the system in caricature would have one believe on the one hand “OERI does 
science and the Secretary does politics” or on the other hand “the Secretary grapples with tough 
policy choices and OERI does research work of minor consequence.” Neither, of course, 
reflected reality, but the mind set degraded institutional performance. 

The advent of strategic plans, performance measures and an analytic agenda provide the 
basis for a much stronger, productive process if the stakeholders have the will to exploit it. 

Turning to OERI, there are collections of structural and process issues that impede the 
achievement of an effective RD&D system. While the 1994 reauthorization and policy direction 
made some matters better, it has altered other problems not at all or made them worse, 
particularly in interaction with constrained staff and financial resources. 

• As noted in earlier papers, allocation to field initiated work is much improved, and 
some concrete steps have been taken on quality and standards of evidence. 

• The effort or admonition to strengthen basic research has failed, and it would seem 
unrealistic to try to “repot” work now done elsewhere (e.g., NICHD) into OERI. 
That does leave open, however, a substantial set of work - basic and applied - to 
translate the cognitive science into classroom settings. 

• Without more staff and financial resources, the institute structure essentially 
reassembled the resources from one organization into five. Whatever has been 
gained in conceptual clarity has been lost in “subcritical mass” units and program 
coherence. Further, the allocation rules continue to concentrate attention on who is 
doing the work rather than what knowledge is being built. Either resources should 
be expanded quickly, probably with relaxation of the allocation rules, or the mission 
narrowed and the institute structure abandoned. 

• The role of the regional laboratories should also be revisited, though in a framework 
that seeks to maximize their comparative advantage to the total knowledge-building 
enterprise. These labs are in a good position to supply both technical assistance and 
hands-on applied research. The Board has received presentations on the need for 
new intermediaries in comprehensive reform efforts which combine applied research 
and technical assistance to help school districts cope with the complex and inter- 
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connected issues in comprehensive reform. The regional labs are ideally situated for 
such work, and in many cases, are doing it. What is troubling about the regional 
laboratories is the continuing tendency to load national missions upon regional 
missions upon local technical assistance without appropriate resources to meet all 
objectives, which can result in an endless and unproductive series of “projects de 
jour.” The important need to focus and make realistic their missions should 
capitalize on the inherent strengths of their niche. 

• OERI and the Board also need to clarify how to weld national coherence into the 
stream of individual projects, which has yet to be accomplished and requires a 
deliberate strategy. This outcome, which is within reach, needs to be predicated on 
a different philosophy than abhorrence of merit-based political processes and ED 
leadership. A national knowledge-building agenda in education will surely have 
content that goes beyond the current Administration’s policy crises, a point that 
seems plausibly understood by all concerned. OERI will need to continue a strong 
voice in research and evidence-based work of lasting significance. But an effort to 
build an effective strategy independent of the Department is doomed to failure. 

Three other matters of stated concern to the Board need to be addressed in a discussion of 
structure and process: the conduct of demonstrations; fast turn-around, policy related work and 
evaluations; and dissemination activities. 

• Demonstrations. As the second Board background paper makes clear, there are 
substantial demonstration activities not only in ORAD within OERI but elsewhere 
in the Department, particularly in the Title I Elementary and Secondary Education 
program. This scattering of demonstration funding throughout ED is a common 
occurrence in all domestic departments. The two issues which such practice raises 
are: (1) whether the demonstrations contain a research or evaluation component; and 
(2) whether they reflect a sensible and coordinated knowledge-building strategy. 

ED is making commendable progress toward assuring few, if any, demonstrations 
without some knowledge-building. ED is doing less well with the second issue, but 
has an opportunity to improve rapidly through the strategic plan and development of 
an analytic agenda. Of particular concern in the area of demonstrations is the 
absence of a federal presence in the comprehensive reform initiatives noted in the 
September 1 997 Board workshop. 

• Policy Short-term Assignments and Evaluation. Questions are often raised as to 
how much, if any, very short-term, high policy content work research organizations 
should undertake and where evaluation work fits in. With respect to the first 
question, most domestic federal agencies concentrate such work in the Secretary’s 
planning and evaluation shop with some involvement of the analytic offices of the 
assistant secretaries. Those offices have, as part of their mission, the translation of 
what we know to what we do. That work is a mixed blessing: important, but 
potentially all consuming. The axiom holds that short-term policy crises will 
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routinely drive out long term knowledge-building (See Morrill & Weiss, 1997). 
Accordingly, there is a strong reason to minimize the involvement of research 
offices in such work. This is not to say that research offices should not contribute 
their knowledge base to policy deliberations and be fully engaged in planning the 
knowledge-building agenda. 

With respect to evaluation, it should be a full partner in the knowledge-building 
agenda. The evaluation planning and execution is now under PES management and 
shared between PES and other assistant secretaries in its execution, depending on 
project focus and staff capacity. For example, PES and ORAD share the evaluation 
projects on ORAD demonstrations. The evaluation plans are now well identified in 
ED strategic plans. This process appears sensible and generally under control. 

• Dissemination. Table 1 in the paper The Needs, Demand for and Supply of 
Educational RD&D identifies dissemination as 25% of total RD&D 1997 
obligations inclusive of NCES and 17% excluding NCES. However, dissemination 
without NCES as a percentage of total R&D obligations is 54%. There are no easy 
rules as to what constitutes a proper percentage; however, the first set of percentages 
do not appear extraordinary, while the relationship to total R&D appear large. We, 
however, are not now in a position to make an unqualified judgment. Since our 
definition of dissemination includes technical assistance, we are not troubled with 
the dispersion of dissemination across various offices. The issue that does raise 
concern is the modality of the dissemination. Most of the dissemination efforts are 
keeping up with technology, but the information is often unsorted for quality, and it 
is usually available passively. The Board’s September 1997 workshop starkly raised 
the issue of the utility of passive information systems in support of comprehensive 
reform efforts, suggesting proactive technical assistance as a more effective model. 
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III. Implications for the Congressional Report 

This review of the federal role and Departmental structure issues together with earlier 
papers present substantial problems for potential inclusion in the Board’s Congressional report. 
The authorization legislation in 1994, and what has followed in implementation, has resolved a 
few problems, but a far larger number remain, some fundamental in nature. 

The central matter to which the analysis of the federal role drives us is whether the 
country and its political leadership believes that it has a serious enough educational problem on 
its hands to which new knowledge can help contribute to a solution. If so or potentially so, then 
the kinds of changes in resources, process and structure considered necessary appear plausible. If 
there is not an important triggering event such as clear recognition of sharp improvement in 
educational performance as a high national priority, it will be difficult at a minimum to change 
the status quo. Plausibility will require crafting a sequential strategy, probably around the new 
money in the OERI budget, to stimulate a series of changes in practice. This strategy would 
necessarily include: (1) a dramatic demonstration of the problem and the nation’s stake in its 
solution; (2) a step-by-step presentation of where new knowledge is needed and can be acquired; 
and (3) a clear demonstration of how the research is science-based rather them faddism. 

With respect to matters of structure and process, the implication of the analysis for the 
reports includes: 

• Whether the Board wants to advocate an integrated RD&D planning process in 
which the Departmental leadership and the political process are treated as partners 
and the source of resources or the enemy? 

• Whether the Board wants to take the NRC judgment seriously about matching 
resources and mission and either get more resources or cut scope and an untenable 
group of underfunded, understaffed institutes? 

• Whether the Board wants to advocate a tighter resource-mission fit for the regional 
laboratories? 

• How the Board wants to deal with the absence of basic research in ED? 

• Whether the Board wants to advocate a new form of intermediaries and a different, 
proactive approach to dissemination? 
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